49

I have a problem to parse 1000's of text files(around 3000 lines in each file of ~400KB size ) in a folder. I did read them using readlines,

   for filename in os.listdir (input_dir) :
       if filename.endswith(".gz"):
          f = gzip.open(file, 'rb')
       else:
          f = open(file, 'rb')

       file_content = f.readlines()
       f.close()
   len_file = len(file_content)
   while i < len_file:
       line = file_content[i].split(delimiter) 
       ... my logic ...  
       i += 1  

This works completely fine for sample from my inputs (50,100 files) . When I ran on the whole input more than 5K files, the time-taken was nowhere close to linear increment.I planned to do an performance analysis and did a Cprofile analysis. The time taken for the more files in exponentially increasing with reaching worse rates when inputs reached to 7K files.

Here is the the cumulative time-taken for readlines , first -> 354 files(sample from input) and second -> 7473 files (whole input)

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 354    0.192    0.001    **0.192**    0.001 {method 'readlines' of 'file' objects}
 7473 1329.380    0.178  **1329.380**    0.178 {method 'readlines' of 'file' objects}

Because of this, the time-taken by my code is not linearly scaling as the input increases. I read some doc notes on readlines(), where people has claimed that this readlines() reads whole file content into memory and hence generally consumes more memory compared to readline() or read().

I agree with this point, but should the garbage collector automatically clear that loaded content from memory at the end of my loop, hence at any instant my memory should have only the contents of my currently processed file right ? But, there is some catch here. Can somebody give some insights into this issue.

Is this an inherent behavior of readlines() or my wrong interpretation of python garbage collector. Glad to know.

Also, suggest some alternative ways of doing the same in memory and time efficient manner. TIA.

4
  • 5
    As a side note, there is never a good reason to write len_file = len(file_content), then a while( i < len_file ): loop with i += 1 and file_content[i] inside. Just use for line in file_content:. If you also need i for something else, use for i, line in enumerate(file_content). You're making things harder for yourself and your readers (and for the interpreter, which means your code may run slower, but that's usually much less important here).
    – abarnert
    Commented Jun 22, 2013 at 1:17
  • Thanks @abarnert. I'll change them .
    – Learner
    Commented Jun 22, 2013 at 1:28
  • 4
    One last style note: In Python, you can just write if filename.endswith(".gz"):; you don't need parentheses around the condition, and shouldn't use them. One of the great things about Python is how easy it is both to skim quickly and to read in-depth, but putting in those parentheses makes it much harder to skim (because you have to figure out whether there's a multi-line expression, a tuple, a genexp, or just code written by a C/Java/JavaScript programmer).
    – abarnert
    Commented Jun 24, 2013 at 18:10
  • Nice tip,duly noted. Will change them as well.
    – Learner
    Commented Jun 24, 2013 at 23:35

2 Answers 2

105

The short version is: The efficient way to use readlines() is to not use it. Ever.


I read some doc notes on readlines(), where people has claimed that this readlines() reads whole file content into memory and hence generally consumes more memory compared to readline() or read().

The documentation for readlines() explicitly guarantees that it reads the whole file into memory, and parses it into lines, and builds a list full of strings out of those lines.

But the documentation for read() likewise guarantees that it reads the whole file into memory, and builds a string, so that doesn't help.


On top of using more memory, this also means you can't do any work until the whole thing is read. If you alternate reading and processing in even the most naive way, you will benefit from at least some pipelining (thanks to the OS disk cache, DMA, CPU pipeline, etc.), so you will be working on one batch while the next batch is being read. But if you force the computer to read the whole file in, then parse the whole file, then run your code, you only get one region of overlapping work for the entire file, instead of one region of overlapping work per read.


You can work around this in three ways:

  1. Write a loop around readlines(sizehint), read(size), or readline().
  2. Just use the file as a lazy iterator without calling any of these.
  3. mmap the file, which allows you to treat it as a giant string without first reading it in.

For example, this has to read all of foo at once:

with open('foo') as f:
    lines = f.readlines()
    for line in lines:
        pass

But this only reads about 8K at a time:

with open('foo') as f:
    while True:
        lines = f.readlines(8192)
        if not lines:
            break
        for line in lines:
            pass

And this only reads one line at a time—although Python is allowed to (and will) pick a nice buffer size to make things faster.

with open('foo') as f:
    while True:
        line = f.readline()
        if not line:
            break
        pass

And this will do the exact same thing as the previous:

with open('foo') as f:
    for line in f:
        pass

Meanwhile:

but should the garbage collector automatically clear that loaded content from memory at the end of my loop, hence at any instant my memory should have only the contents of my currently processed file right ?

Python doesn't make any such guarantees about garbage collection.

The CPython implementation happens to use refcounting for GC, which means that in your code, as soon as file_content gets rebound or goes away, the giant list of strings, and all of the strings within it, will be freed to the freelist, meaning the same memory can be reused again for your next pass.

However, all those allocations, copies, and deallocations aren't free—it's much faster to not do them than to do them.

On top of that, having your strings scattered across a large swath of memory instead of reusing the same small chunk of memory over and over hurts your cache behavior.

Plus, while the memory usage may be constant (or, rather, linear in the size of your largest file, rather than in the sum of your file sizes), that rush of mallocs to expand it the first time will be one of the slowest things you do (which also makes it much harder to do performance comparisons).


Putting it all together, here's how I'd write your program:

for filename in os.listdir(input_dir):
    with open(filename, 'rb') as f:
        if filename.endswith(".gz"):
            f = gzip.open(fileobj=f)
        words = (line.split(delimiter) for line in f)
        ... my logic ...  

Or, maybe:

for filename in os.listdir(input_dir):
    if filename.endswith(".gz"):
        f = gzip.open(filename, 'rb')
    else:
        f = open(filename, 'rb')
    with contextlib.closing(f):
        words = (line.split(delimiter) for line in f)
        ... my logic ...
6
  • I should have told this earlier. My inputs directory might contain gzip file and also normal text file - so for file open i'am using a if else construct. I'm afraid this 'with' might not work out.
    – Learner
    Commented Jun 22, 2013 at 1:10
  • @Learner: Sure it will: with open('foo', 'rb') as f:, then you can create a GzipFile(fileobj=f) if necessary (or an io.IOTextWrapper if it's a text file you want decoded to unicode, or a csv.reader if it's a CSV file you want decoded to rows, etc.). At any rate, the with part isn't relevant here; all of the options are exactly the same options with explicit close, except more verbose and less robust.
    – abarnert
    Commented Jun 22, 2013 at 1:13
  • I'm not sure if I understood the iotextwrapper part. Any links to follow ? TIA :)
    – Learner
    Commented Jun 22, 2013 at 1:34
  • 1
    @Learner: I'm assuming you're using Python 2, yes? If so, the reference docs are here, and the way to learn is… read the differences between Python 2 text files and Python 3 text files (maybe start here); io.TextIOWrapper turns the former into the latter, so you can write clean Py3-style code that only deals with unicode objects, not encoded bytes, even in Py2.
    – abarnert
    Commented Jun 22, 2013 at 1:43
  • 1
    @Learner: Glad it helped. closing isn't useful that often—most of the time, you've just got a file or something else that can be used directly in a with statement—but it is handy to know for cases like this. Anyway, the important part (the part that sped up your code) is using the file (or GzipFile) directly as an iterable, instead of readline()-ing the whole thing into memory to use the list as an iterable, as Óscar López explained before me.
    – abarnert
    Commented Jun 24, 2013 at 18:06
19

Read line by line, not the whole file:

for line in open(file_name, 'rb'):
    # process line here

Even better use with for automatically closing the file:

with open(file_name, 'rb') as f:
    for line in f:
        # process line here

The above will read the file object using an iterator, one line at a time.

17
  • 1
    That readlines will read the whole file at once into a list, which can be a problem if it's big - it'll use a lot of memory! Commented Jun 22, 2013 at 0:55
  • 1
    Yes, you'll be consuming memory and eventually you'll start paging into disk if the physical memory runs out. And no, the GC is not deterministic, so you can't tell when the memory is going to be freed - in fact, part of the reasons for the slowdown could be the GC running Commented Jun 22, 2013 at 1:05
  • 1
    @ÓscarLópez: docs.python.org/2/c-api/intro.html#reference-counts documents how the refcounting works. (The documentation on cycle breaking is elsewhere, but not relevant here.) The proof that it's deterministic is trivial: a pure refcounting GC is deterministic by definition (and a refcounting-plus-cycle-breaking GC is likewise deterministic when there are no cycles).
    – abarnert
    Commented Jun 22, 2013 at 1:11
  • 3
    @ÓscarLópez: The whole point of refcounting is that it doesn't have to run. Every time a reference goes away (e.g., a name is rebound or goes out of scope), the count on the referenced object is decreased, and if it reaches 0, the object is reclaimed immediately. (The cycle detector is another, more complicated story, but again, it's not relevant here, because there are no cycles in the OP's code.) The Wikipedia article explains it pretty well.
    – abarnert
    Commented Jun 22, 2013 at 1:20
  • 3
    @abarnert thanks for clarifying that, I learnt something new :) Commented Jun 22, 2013 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.