0

I have few thousands of text files where each file is of following form:

Some Key: value1
Some Other Key: value2
Another Key: value3
... < 300+ such entries > ...

I want to read each of these files as dictionary and populate as a row in pandas dataframe. Though at this moment I am not sure if all the files have exact same keys or not but I'm hoping that there will not be too much variations as these files are logs from some tool.

What is the easiest way to read each file as dictionary so that it's correct by constructions? As of now, my simple code is like following:

with open(log_data_file, mode="r") as txt_file:
    for line in txt_file:
        keyval = line.strip().split(sep=":", maxsplit=3)
        
        if len(keyval) != 2:
            # some debug print
            continue
        
        data[keyval[0]] = keyval[1]
        

Possibly, I can add some logic to handle a line if that satisfies a regular expression. But beyond that, is there package in python where I can specify the grammar for the file and handle [iterate over list of (key, value)] the file only if the grammar is satisfied and file read is successful?

4
  • 2
    Your code is the correct way to do it.
    – Barmar
    Commented Jan 19, 2023 at 23:26
  • Thanks, since I'm not that expert in Python, I'm checking if there's any better/easier way to do it.
    – soumeng78
    Commented Jan 19, 2023 at 23:50
  • There are parsers for standard file formats, like YAML, JSON, .INI, etc. This is not a standard config file, I don't think there's anything prewritten for it. It's so simple that it's not work bothering to write a library.
    – Barmar
    Commented Jan 19, 2023 at 23:53
  • 1
    Your example file shows a space after the comma, you might want to strip that from the value. data[keyval[0]] = keyval[1].strip()
    – wwii
    Commented Jan 20, 2023 at 0:01

1 Answer 1

1
import pandas
import re

read_buffer = """
Some Key: value1
Some Other Key: value2
Another Key: value3
... < 300+ such entries > ...
"""
regex = r"(?P<key>.+):\s(?P<value>\S+)"
matches = re.finditer(regex, read_buffer)
records = [match.groupdict() for match in matches]
df = pd.DataFrame(records)
df = df.T.reset_index()
df.columns = df.iloc[0]
df = df[1:]
df

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.