I may have found a reason why write
is slower than writelines
. In looking through the CPython source (3.4.3) I found the code for the write
function (took out irrelevent parts).
Modules/_io/fileio.c
static PyObject *
fileio_write(fileio *self, PyObject *args)
{
Py_buffer pbuf;
Py_ssize_t n, len;
int err;
...
n = write(self->fd, pbuf.buf, len);
...
PyBuffer_Release(&pbuf);
if (n < 0) {
if (err == EAGAIN)
Py_RETURN_NONE;
errno = err;
PyErr_SetFromErrno(PyExc_IOError);
return NULL;
}
return PyLong_FromSsize_t(n);
}
If you notice, this function actually returns a value, the size of the string that has been written, which is another function call.
I tested this out to see if it actually had a return value, and it did.
with open('test.txt', 'w+') as f:
x = f.write("hello")
print(x)
>>> 5
The following is the code for the writelines
function implementation in CPython (took out irrelevent parts).
Modules/_io/iobase.c
static PyObject *
iobase_writelines(PyObject *self, PyObject *args)
{
PyObject *lines, *iter, *res;
...
while (1) {
PyObject *line = PyIter_Next(iter);
...
res = NULL;
do {
res = PyObject_CallMethodObjArgs(self, _PyIO_str_write, line, NULL);
} while (res == NULL && _PyIO_trap_eintr());
Py_DECREF(line);
if (res == NULL) {
Py_DECREF(iter);
return NULL;
}
Py_DECREF(res);
}
Py_DECREF(iter);
Py_RETURN_NONE;
}
If you notice, there is no return value! It simply has Py_RETURN_NONE
instead of another function call to calculate the size of the written value.
So, I went ahead and tested that there really wasn't a return value.
with open('test.txt', 'w+') as f:
x = f.writelines(["hello", "hello"])
print(x)
>>> None
The extra time that write
takes seems to be due to the extra function call taken in the implementation to produce the return value. By using writelines
, you skip that step and the fileio is the only bottleneck.
Edit: write
documentation
writelines
might join the list of strings with newlines and write them all at once. I doubt thatwritelines
callswrite
for every element in the list/generator. I would assume that the speed increase comes from the implmentation in C.w.writelines(ls)
withw.write("\n".join(ls))
? How does the speed compare to your existing cases?len(ls) == 100000:
so potentially you write less lines to one file, alsoopen("otherfile.txt", "w",buffering=1000) as w:
beats writelines for me