10

A while ago, I made a Python script which looked similar to this:

with open("somefile.txt", "r") as f, open("otherfile.txt", "a") as w:
    for line in f:
        w.write(line)

Which, of course, worked pretty slowly on a 100mb file.

However, I changed the program to do this

ls = []
with open("somefile.txt", "r") as f, open("otherfile.txt", "a") as w:
    for line in f:
        ls.append(line)
        if len(ls) == 100000:
            w.writelines(ls)
            del ls[:]

And the file copied much faster. My question is, why does the second method work faster even though the program copies the same number of lines (albeit collects them and prints them one by one)?

18
  • 2
    This is interesting. I think it may have something to do with the IO operations. writelines might join the list of strings with newlines and write them all at once. I doubt that writelines calls write for every element in the list/generator. I would assume that the speed increase comes from the implmentation in C.
    – Brobin
    Jul 27, 2015 at 15:46
  • Fewer hard drive head seeks between reading and writing? Jul 27, 2015 at 15:48
  • 2
    What if you replaced w.writelines(ls) with w.write("\n".join(ls))? How does the speed compare to your existing cases?
    – Kevin
    Jul 27, 2015 at 15:55
  • 3
    Your logic is also slightly flawed as you only write when len(ls) == 100000: so potentially you write less lines to one file, also open("otherfile.txt", "w",buffering=1000) as w: beats writelines for me Jul 27, 2015 at 16:19
  • 1
    What's your python version?
    – Mazdak
    Jul 27, 2015 at 16:30

3 Answers 3

2

I may have found a reason why write is slower than writelines. In looking through the CPython source (3.4.3) I found the code for the write function (took out irrelevent parts).

Modules/_io/fileio.c

static PyObject *
fileio_write(fileio *self, PyObject *args)
{
    Py_buffer pbuf;
    Py_ssize_t n, len;
    int err;
    ...
    n = write(self->fd, pbuf.buf, len);
    ...

    PyBuffer_Release(&pbuf);

    if (n < 0) {
        if (err == EAGAIN)
            Py_RETURN_NONE;
        errno = err;
        PyErr_SetFromErrno(PyExc_IOError);
        return NULL;
    }

    return PyLong_FromSsize_t(n);
}

If you notice, this function actually returns a value, the size of the string that has been written, which is another function call.

I tested this out to see if it actually had a return value, and it did.

with open('test.txt', 'w+') as f:
    x = f.write("hello")
    print(x)

>>> 5

The following is the code for the writelines function implementation in CPython (took out irrelevent parts).

Modules/_io/iobase.c

static PyObject *
iobase_writelines(PyObject *self, PyObject *args)
{
    PyObject *lines, *iter, *res;

    ...

    while (1) {
        PyObject *line = PyIter_Next(iter);
        ...
        res = NULL;
        do {
            res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL);
        } while (res == NULL && _PyIO_trap_eintr());
        Py_DECREF(line);
        if (res == NULL) {
            Py_DECREF(iter);
            return NULL;
        }
        Py_DECREF(res);
    }
    Py_DECREF(iter);
    Py_RETURN_NONE;
}

If you notice, there is no return value! It simply has Py_RETURN_NONE instead of another function call to calculate the size of the written value.

So, I went ahead and tested that there really wasn't a return value.

with open('test.txt', 'w+') as f:
    x = f.writelines(["hello", "hello"])
    print(x)

>>> None

The extra time that write takes seems to be due to the extra function call taken in the implementation to produce the return value. By using writelines, you skip that step and the fileio is the only bottleneck.

Edit: write documentation

11
  • How exactly does returning length of string make such a difference? I mean, if you run a regular return len(line) it is instantaneous!
    – rassa45
    Jul 27, 2015 at 18:13
  • It seems instantaneous, but compounded thousands of times, it might take a while. Also, returning the length uses more memory.
    – Brobin
    Jul 27, 2015 at 18:13
  • The order of len() is O(1) so i don't think that makes any problem here!
    – Mazdak
    Jul 27, 2015 at 18:30
  • I realize that its O(1) but that doesn't mean that that calculation can not be a source of the slowdown. O(1) means it is computed in linear time. It still takes time to compute!
    – Brobin
    Jul 27, 2015 at 18:33
  • Would it be a good idea to test the write source code with and without the line length calculations?
    – rassa45
    Jul 28, 2015 at 2:54
0

I do not agree with the other answer here.

It is simply a coincidence. It highly depends on your environment:

  • What OS?
  • What HDD/CPU?
  • What HDD file system format?
  • How busy is your CPU/HDD?
  • What Python version?

Both pieces of code do the absolute same thing with tiny differences in performance.

For me personally .writelines() takes longer to execute then your first example using .write(). Tested with 110MB text file.

I will not post my machine specs on purpose.

Test .write(): ------copying took 0.934000015259 seconds (dashes for readability)

Test .writelines(): copying took 0.936999797821 seconds

Also tested with small and as large as 1.5GB files with the same results. (writelines always beeing slightly slower, up to 0.5sec difference for 1.5GB file).

4
  • Maybe if you use longer lines this will take more time
    – rassa45
    Jul 28, 2015 at 2:57
  • I think the number of lines makes a difference as well. And if you have a heavenly processor, well, obviously you are getting results :)
    – rassa45
    Jul 28, 2015 at 2:58
  • Last thing you could do is change the number of lines collected to 1000000 or something
    – rassa45
    Jul 28, 2015 at 2:59
  • I've made a number of different test including short/long lines. My whole point was it is very environment specific rather than python implementation (algorithm) specific.
    – SDekov
    Jul 28, 2015 at 8:41
-1

That's because of that in first part you have to call the method write for all the lines in each iteration which makes your program take much time to run. But in second code although your waste more memory but it performs better because you have called the writelines() method each 100000 line.

Let see this is source,this is the source of writelines function :

def writelines(self, list_of_data):
    """Write a list (or any iterable) of data bytes to the transport.

    The default implementation concatenates the arguments and
    calls write() on the result.
    """
    if not _PY34:
        # In Python 3.3, bytes.join() doesn't handle memoryview.
        list_of_data = (
            bytes(data) if isinstance(data, memoryview) else data
            for data in list_of_data)
    self.write(b''.join(list_of_data))

As you can see it joins all the list items and calls the write function one time.

Note that joining the data here takes time but its less than the time for calling the write function for each line.But since you use python 3.4 in ,it writes the lines one at a time rather than joining them so it would be much faster than write in this case :

  • cStringIO.writelines() now accepts any iterable argument and writes the lines one at a time rather than joining them and writing once. Made a parallel change to StringIO.writelines(). Saves memory and makes suitable for use with generator expressions.
9
  • Yes, that's what the code is doing, but you didn't expain why the second method is faster.
    – Brobin
    Jul 27, 2015 at 15:58
  • But surely writelines does more work than write would, so you can't just say "it's better to use the approach that has fewer function calls"
    – Kevin
    Jul 27, 2015 at 15:58
  • @Brobin Yes I'm looking for the reason in source!
    – Mazdak
    Jul 27, 2015 at 16:03
  • @Kevin Indeed I would update the answer with that reason!
    – Mazdak
    Jul 27, 2015 at 16:04
  • So is the time taken up by opening the file and closing it multiple times? And this does not occur with smaller files because you don't open and close as much?
    – rassa45
    Jul 27, 2015 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.