2

I have log files that contain a a date/time with a varied # of lines between the next date/time

ex.

time-date
2/07/18 13:55:00.983

msecVal = pyparsing.Word(pyparsing.nums, max=3)
numPair = pyparsing.Word(pyparsing.nums, exact=2)
dateStr = pyparsing.Combine(numPair + '/' + numPair + '/' + numPair)

timeString = pyparsing.Combine(numPair + ':' + numPair + ':' +     numPair\               
       + '.' + msecVal)

log file will be

time:date:  line of text
    possible 2nd line of text
    possible 3rd line of text...
    time:date:  line of text
time:date: line of text
    possible 2nd line of text
    possible 3rd line of text...
    possible <n> line of text...
time:date:  line of text

Input will be a large text log file in the above format. I'd like to produce a list list of grouped elements

[[time],[all text until next time]],[[time],[all text until next time]...

I can do this if each time/date entry were a single line.. it's spanning between a random # of multiple lines until the next time/date entry I'm having problems with.

1 Answer 1

0

Here is how I interpret your definition of a log enty:

"A date-time at the beginning of a line, followed by a colon, followed by everything up until the next date-time at the beginning of a line, even if there might be date-times embedded in the line."

There are two pyparsing features that you need to solve this:

  • LineStart - to distinguish date-times at the start of the line vs those in the body of the line

  • SkipTo - quick way to skip over unstructured text until a matching expression is found

I added these expressions to your code (I imported pyparsing as 'pp' because I am a lazy typist):

dateTime = dateStr + timeString

# log entry date-time keys only match if they are at the start of the line
dateTimeKey = pp.LineStart() + dateTime

# define a log entry as a date-time key, followed by everything up to the next 
# date-time key, or to the end of the input string
# (use results names to make it easy to get at the parts of the log entry)
logEntry = pp.Group(dateTimeKey("time") + ':' + pp.Empty()
                    + pp.SkipTo(dateTimeKey | pp.StringEnd())("body"))

I converted your sample to have different date times in it for testing, and we get this:

sample = """\
2/07/18 13:55:00.983:  line of text
    possible 2nd line of text
    possible 3rd line of text...
    2/07/19 13:55:00.983:  line of text
2/07/20 13:55:00.983: line of text
    possible 2nd line of text
    possible 3rd line of text...
    possible <n> line of text...
2/07/21 13:55:00.983:  line of text
"""

print(pp.OneOrMore(logEntry).parseString(sample).dump())

Gives:

[['2/07/18', '13:55:00.983', ':', 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    2/07/19 13:55:00.983:  line of text'], ['2/07/20', '13:55:00.983', ':', 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    possible <n> line of text...'], ['2/07/21', '13:55:00.983', ':', 'line of text']]
[0]:
  ['2/07/18', '13:55:00.983', ':', 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    2/07/19 13:55:00.983:  line of text']
  - body: 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    2/07/19 13:55:00.983:  line of text'
  - time: ['2/07/18', '13:55:00.983']
[1]:
  ['2/07/20', '13:55:00.983', ':', 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    possible <n> line of text...']
  - body: 'line of text\n    possible 2nd line of text\n    possible 3rd line of text...\n    possible <n> line of text...'
  - time: ['2/07/20', '13:55:00.983']
[2]:
  ['2/07/21', '13:55:00.983', ':', 'line of text']
  - body: 'line of text'
  - time: ['2/07/21', '13:55:00.983']

I also had to convert your num_pair to:

numPair = pp.Word(pp.nums, max=2)

else it would not match the leading single-digit '2' in your sample date.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.