12

In an interview, an interviewer asked me why readline() is much slower than readlines() in Python?

I answered that readlines() need to read many times, which need more spending.

I don't know whether my answer is correct.

If my answer is correct, what is the spending?

Thanks!

PS: I know the difference between readline() and readlines()!!!

readlines() read the whole file in one time, and readline() can read only one line.

I hope somebody can tell me some knowledge about IO of the file from the disk.

1
  • 1
    Welcome to StackOverflow! Did you read the documentation?
    – llrs
    Commented Apr 8, 2014 at 16:21

2 Answers 2

24

Just for fun, I wrote a bunch of functions that iterate over a file and put each line into a list:

#!/usr/bin/python

def readlines():
    with open("sorted_output.txt") as f:
        line = f.readlines()

def readline():
    with open("sorted_output.txt") as f:
        line = f.readline()
        lines = []
        while line:
            lines.append(line)
            line = f.readline()

def iterate():
    with open("sorted_output.txt") as f:
        lines = []
        for line in f:
            lines.append(line)

def comprehension():
    with open("sorted_output.txt") as f:
        lines = [line for line in f]

Here's is how each of them performed on a file with 69,073 lines, using Python 2.6 (note, these results may be different on newer versions of Python):

dano@hostname:~> python -mtimeit -s 'import test' 'test.readline()'
10 loops, best of 3: 78.3 msec per loop
dano@hostname:~> python -mtimeit -s 'import test' 'test.readlines()'
10 loops, best of 3: 21.6 msec per loop
dano@hostname:~> python -mtimeit -s 'import test' 'test.comprehension()'
10 loops, best of 3: 23.6 msec per loop
dano@hostname:~> python -mtimeit -s 'import test' 'test.iterate()'
10 loops, best of 3: 33.3 msec per loop

So, readlines() is the fastest here, though iterating over each line using a list comprehension almost matches it. My guess is the speed differences between each approach is mostly the result of the high overhead of function calls in Python (the more function calls required, the slower the approach), but there may be other factors, as well. Hopefully someone more knowledgeable than me can comment on that.

In addition to performance, one other important consideration when deciding which of these methods to use is memory cost. Using readlines() will read the entire file into memory at once. If you're dealing with a huge file, it could cause serious performance issues or crash the program altogether if you were to try to read the entire thing into memory at once. In those cases, you'd want to use the approach in iterate(), since it only reads one line into memory at a time. In cases where you're just doing some kind of processing on each line and then throwing it away, this is usually the way to go, even if it is slightly slower than readlines(), because you don't take the same memory hit. Of course, if your goal in the end is to store the entire file in a Python list, you're going to pay that memory cost anyway, so readlines() will work fine.

0
-1

From the documentation, for file.readline()

f.readline() reads a single line from the file; a newline character (\n) is left at the end of the string, and is only omitted on the last line of the file if the file doesn’t end in a newline.

Read one entire line from the file. A trailing newline character is kept in the string (but may be absent when a file ends with an incomplete line). If the size argument is present and non-negative, it is a maximum byte count (including the trailing newline) and an incomplete line may be returned. When size is not 0, an empty string is returned only when EOF is encountered immediately.

And for the readlines()

If you want to read all the lines of a file in a list you can also use list(f) or f.readlines().

Read until EOF using readline() and return a list containing the lines thus read. If the optional sizehint argument is present, instead of reading up to EOF, whole lines totalling approximately sizehint bytes (possibly after rounding up to an internal buffer size) are read. Objects implementing a file-like interface may choose to ignore sizehint if it cannot be implemented, or cannot be implemented efficiently.

So the readlines uses the readline function to "read many times" the file. Other answers can answer more in detail the performance of each one.

1
  • 9
    The question isn't how are they different, but why the performance between the two are different. Commented Apr 8, 2014 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.