12

I'm working on a Python script to go through two files - one containing a list of UUIDs, the other containing a large amount of log entries - each line containing one of the UUIDs from the other file. The purpose of the program is to create a list of the UUIDS from file1, then for each time that UUID is found in the log file, increment the associated value for each time a match is found.

So long story short, count how many times each UUID appears in the log file. At the moment, I have a list which is populated with UUID as the key, and 'hits' as the value. Then another loop which iterates over each line of the log file, and checking if the UUID in the log matches a UUID in the UUID list. If it matches, it increments the value.

    for i, logLine in enumerate(logHandle):         #start matching UUID entries in log file to UUID from rulebase
        if logFunc.progress(lineCount, logSize):    #check progress
            print logFunc.progress(lineCount, logSize)  #print progress in 10% intervals
        for uid in uidHits:
            if logLine.count(uid) == 1:             #for each UUID, check the current line of the log for a match in the UUID list
                uidHits[uid] += 1                   #if matched, increment the relevant value in the uidHits list
                break                                #as we've already found the match, don't process the rest
        lineCount += 1               

It works as it should - but I'm sure there is a more efficient way of processing the file. I've been through a few guides and found that using 'count' is faster than using a compiled regex. I thought reading files in chunks rather than line by line would improve performance by reducing the amount of disk I/O time but the performance difference on a test file ~200MB was neglible. If anyone has any other methods I would be very grateful :)

5
  • 2
    File I/O is usually buffered regardless of the size of chunks you actually read.
    – user395760
    Jun 2, 2011 at 13:59
  • 3
    Does it need to be more efficient? How long does it take? How long do you need it to take? You may well have already hit the performance limit of your storage (disk), in which case it doesn't matter how much faster your Python script is. Jun 2, 2011 at 14:05
  • It's running through a test file now - it's half way through a 10GB file and it's taken about 30mins. Being my first Python outing I don't really know if that's fast or slow. There's no requirement for it to complete in x minutes, but faster is better ;)
    – SG84
    Jun 2, 2011 at 14:09
  • In your example, the second if statement is empty (no indented code after that). Could you fix it? Jun 2, 2011 at 14:15
  • 4
    You're going over all the UIDs for every line in the file. Instead, find the UUID in every line and look it up in a dictionary. Try to do as little as possible in the part of the code that's getting called most often. Jun 2, 2011 at 14:25

5 Answers 5

14

Think functionally!

  1. Write a function which will take a line of the log file and return the uuid. Call it uuid, say.

  2. Apply this function to every line of the log file. If you are using Python 3 you can use the built-in function map; otherwise, you need to use itertools.imap.

  3. Pass this iterator to a collections.Counter.

    collections.Counter(map(uuid, open("log.txt")))
    

This will be pretty much optimally efficient.

A couple comments:

  • This completely ignores the list of UUIDs and just counts the ones that appear in the log file. You will need to modify the program somewhat if you don't want this.

    • Your code is slow because you are using the wrong data structures. A dict is what you want here.
4
  • Thanks for the input - once this test run has finished and I get my resources back I'll take a look. I think I used a list over a dict because I wanted to maintain the order of the UUIDS but I guess I could use the list later on as an index and then pull the corresponding values from the dict?
    – SG84
    Jun 2, 2011 at 14:27
  • 5
    @SG84 you may look to an excellent article on Python's generators dabeaz.com/generators/Generators.pdf, especially for processing large file. You'll be enlighten :-) Jun 2, 2011 at 14:32
  • I did try a generator earlier, after I picked my brain up off the floor and got it working the performance gain wasn't a lot different to just using a for loop over the file. thanks for the link though, it all makes for good reading
    – SG84
    Jun 2, 2011 at 14:41
  • Ok so after getting the hang of generators and the collections module, i've amended a couple of the functions in my code and the total execution time on a 250MB file is down from 96.4 seconds to 5.4!! That's a massive win. Many thanks to all for their input :)
    – SG84
    Jun 6, 2011 at 14:33
3

This is not a 5-line answer to your question, but there was an excellent tutorial given at PyCon'08 called Generator Tricks for System Programmers. There is also a followup tutorial called A Curious Course on Coroutines and Concurrency.

The Generator tutorial specifically uses big log file processing as its example.

3

Like folks above have said, with a 10GB file you'll probably hit the limits of your disk pretty quickly. For code-only improvements, the generator advice is great. In python 2.x it'll look something like

uuid_generator = (line.split(SPLIT_CHAR)[UUID_FIELD] for line in file)

It sounds like this doesn't actually have to be a python problem. If you're not doing anything more complex than counting UUIDs, Unix might be able to solve your problems faster than python can.

cut -d${SPLIT_CHAR} -f${UUID_FIELD} log_file.txt | sort | uniq -c 
3

Have you tried mincemeat.py? It is a Python implementation of the MapReduce distributed computing framework. I'm not sure if you'll have performance gain since I've not yet processed 10GB of data before using it, though you might explore this framework.

0

Try measuring where most time is spent, using a profiler http://docs.python.org/library/profile.html

Where best to optimise will depend on the nature of your data: If the list of uuids isn't very long, you may find, for example, that a large proportion of time is spend on the "if logFunc.progress(lineCount, logSize)". If the list is very long, you it could help to save the result of uidHits.keys() to a variable outside the loop and iterate over that instead of the dictionary itself, but Rosh Oxymoron's suggesting of finding the id first and then checking for it in uidHits would probably help even more.

In any case, you can eliminate the lineCount variable, and use i instead. And find(uid) != -1 might be better than count(uid) == 1 if the lines are very long.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.