9

I'm wondering if anyone has a sort of hacky / cool solution to this problem . I have a text file like so:

NAME:name
ID:id
PERSON:person
LOCATION:location

NAME:name
morenamestuff
ID:id
PERSON:person
LOCATION:location

JUNK

So I have some blocks that all contain lines that can be split into a dict, and some that cannot. How can I take lines without the : character and join them to the previous line? Here's what I'm currently doing

# loop through chunk
    # the first element of dat is a Title, so skip that
    key_map = dict(x.split(':') for x in dat[1:])

But I of course get an error because the second chunk has a line without the : character. So I wanted my dict to look something like this after correctly splitting it:

# there will be a key_map for each chunk of data
key_map['NAME'] == 'name morenamestuff' # 3rd line appended to previous
key_map['ID'] == 'id'
key_map['PERSON'] = 'person'
key_map['LOCATION'] = 'location

Solution

EDIT: Here's my final solution on github, and the full code here:

parseScript.py

import re
import string

bad_chars = '(){}"<>[] '     # characers we want to strip from the string
key_map = []

# parse file
with open("dat.txt") as f:
    data = f.read()
    data = data.strip('\n')
    data = re.split('}|\[{', data)

# format file
with open("format.dat") as f:
    formatData = [x.strip('\n') for x in f.readlines()]

data = filter(len, data)

# strip and split each station
for dat in data[1:-1]:
    # perform black magic, don't even try to understand this
    dat = dat.translate(string.maketrans("", "", ), bad_chars).split(',')
    key_map.append(dict(x.split(':') for x in dat if ':' in x ))
    if ':' not in dat[1]:key_map['NAME']+=dat[k][2]


for station in range(0, len(key_map)):
    for opt in formatData:
        print opt,":",key_map[station][opt]
    print ""

dat.txt

View raw here

format.dat

NAME
STID
LONGITUDE
LATITUDE
ELEVATION
STATE
ID

out.dat

View raw here

0

5 Answers 5

2

When in doubt, write your own generator.

Add in itertools.groupby to chunk by groups of text delimited by whitespace breaks.

def chunker(s):
     it = iter(s)
     out = [next(it)]
     for line in it:
         if ':' in line or not line:
             yield ' '.join(out)
             out = []
         out.append(line)
     if out:
         yield ' '.join(out)

usage:

from itertools import groupby

[dict(x.split(':') for x in g) for k,g in groupby(chunker(lines), bool) if k]
Out[65]: 
[{'ID': 'id', 'LOCATION': 'location', 'NAME': 'name', 'PERSON': 'person'},
 {'ID': 'id',
  'LOCATION': 'location',
  'NAME': 'name morenamestuff',
  'PERSON': 'person'}]

(if those fields are always the same, I'd go with something like creating some namedtuples instead of a bunch of dicts)

from collections import namedtuple

Thing = namedtuple('Thing', 'ID LOCATION NAME PERSON')

[Thing(**dict(x.split(':') for x in g)) for k,g in groupby(chunker(lines), bool) if k]
Out[76]: 
[Thing(ID='id', LOCATION='location', NAME='name', PERSON='person'),
 Thing(ID='id', LOCATION='location', NAME='name morenamestuff', PERSON='person')]
1

Here is something that addresses all your requirements. It handles joining of multiple lines, ignoring blank lines, and ignoring junk lines that do not appear within a block. It is implemented as a generator that yields each dictionary as it is completed.

def parser(data):
    d = {}
    for line in data:
        line = line.strip()
        if not line:
            if d:
                yield d
            d = {}
        else:
            if ':' in line:
                key, value = line.split(':')
                d[key] = value
            else:
                if d:
                    d[key] = '{} {}'.format(d[key], line)
    if d:
        yield d

When run with this data:

ignore me

NAME:name1
ID:id1
PERSON:person1
LOCATION:location1

NAME:name2
morenamestuff
ID:id2
PERSON:person2
LOCATION:location2


junk
and
other
stuff


NAME:name3
morenamestuff
and more
ID:id3
PERSON:person3
more person stuff
LOCATION:location3

JUNK
MORE JUNK
>>> for d in parser(open('data')):
...     print d
{'PERSON': 'person1', 'LOCATION': 'location1', 'NAME': 'name1', 'ID': 'id1'}
{'PERSON': 'person2', 'LOCATION': 'location2', 'NAME': 'name2 morenamestuff', 'ID': 'id2'}
{'PERSON': 'person3 more person stuff', 'LOCATION': 'location3', 'NAME': 'name3 morenamestuff and more', 'ID': 'id3'}

You can grab the lot as a list:

>>> results = list(parser(open('data')))
>>> results
[{'PERSON': 'person1', 'LOCATION': 'location1', 'NAME': 'name1', 'ID': 'id1'}, {'PERSON': 'person2', 'LOCATION': 'location2', 'NAME': 'name2 morenamestuff', 'ID': 'id2'}, {'PERSON': 'person3 more person stuff', 'LOCATION': 'location3', 'NAME': 'name3 morenamestuff and more', 'ID': 'id3'}]
1

I don't find itertools or regex particularly nice to work with, here's a pure-python solution

separator = ':'
output = []
chunk = None

with open('/tmp/stuff.txt') as f:
    for line in (x.strip() for x in f):

        if not line:
            # we are between 'chunks'
            chunk, key = None, None
            continue

        if chunk is None:
            # we are at the beginning of a new 'chunk'
            chunk, key = {}, None
            output.append(chunk)

        if separator in line:
            key, val = line.split(separator)
            chunk[key] = val
        else:
            chunk[key] += line
0

not as elegant, as you requested, but this works

dat=[['NAME:name',
      'ID:id',
      'PERSON:person',
      'LOCATION:location'],
      ['NAME:name',
      'morenamestuff',
      'ID:id',
      'PERSON:person',
      'LOCATION:location']]
k=1
key_map = dict(x.split(':') for x in dat[k] if ':' in x )
if ':' not in dat[k][1]:key_map['NAME']+=dat[k][1]

key_map>>
{'ID': 'id',
'LOCATION': 'location',
'NAME': 'namemorenamestuff',
'PERSON': 'person'}
0
0

Just add something to lines with no ":".

if line.find(':') == -1:
    line=line+':None'

Then you won't get an error.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.