26

I have written the following script to concatenate all the files in the directory into one single file.

Can this be optimized, in terms of

  1. idiomatic python

  2. time

Here is the snippet:

import time, glob

outfilename = 'all_' + str((int(time.time()))) + ".txt"

filenames = glob.glob('*.txt')

with open(outfilename, 'wb') as outfile:
    for fname in filenames:
        with open(fname, 'r') as readfile:
            infile = readfile.read()
            for line in infile:
                outfile.write(line)
            outfile.write("\n\n")
2

6 Answers 6

50

Use shutil.copyfileobj to copy data:

import shutil

with open(outfilename, 'wb') as outfile:
    for filename in glob.glob('*.txt'):
        if filename == outfilename:
            # don't want to copy the output into the output
            continue
        with open(filename, 'rb') as readfile:
            shutil.copyfileobj(readfile, outfile)

shutil reads from the readfile object in chunks, writing them to the outfile fileobject directly. Do not use readline() or a iteration buffer, since you do not need the overhead of finding line endings.

Use the same mode for both reading and writing; this is especially important when using Python 3; I've used binary mode for both here.

4
  • Why is important to use the same mode for writing and reading?
    – Juan David
    Aug 30, 2017 at 15:11
  • 1
    @JuanDavid: because shutil will use .read() calls on one, .write() calls on the other file object, passing the read data from one to the other. If one is open in binary mode, the other in text, you are passing through incompatible data (binary data to a text file, or text data to a binary file).
    – Martijn Pieters
    Aug 30, 2017 at 18:00
  • The code here doesn't work with CSV files, dang. But it did give me some good inspiration for how to accomplish this with CSV. I'm relatively new to Python.
    – brett s
    Feb 23, 2021 at 2:25
  • @bretts: the contents of the files shouldn't matter; perhaps your CSV files are missing the last newline separator, or are using different delimiter formats?
    – Martijn Pieters
    Feb 23, 2021 at 9:24
4

Using Python 2.7, I did some "benchmark" testing of

outfile.write(infile.read())

vs

shutil.copyfileobj(readfile, outfile)

I iterated over 20 .txt files ranging in size from 63 MB to 313 MB with a joint file size of ~ 2.6 GB. In both methods, normal read mode performed better than binary read mode and shutil.copyfileobj was generally faster than outfile.write.

When comparing the worst combination (outfile.write, binary mode) with the best combination (shutil.copyfileobj, normal read mode), the difference was quite significant:

outfile.write, binary mode: 43 seconds, on average.

shutil.copyfileobj, normal mode: 27 seconds, on average.

The outfile had a final size of 2620 MB in normal read mode vs 2578 MB in binary read mode.

2
  • Interesting. What platform was that?
    – ellockie
    Jun 26, 2016 at 6:37
  • I roughly work on two platforms: Linux Fedora 16, different nodes or Windows 7 Enterprise SP1 with an Intel Core(TM)2 Quad CPU Q9550, 2.83 GHz. I think it was the latter. Jul 1, 2016 at 13:17
3

You can iterate over the lines of a file object directly, without reading the whole thing into memory:

with open(fname, 'r') as readfile:
    for line in readfile:
        outfile.write(line)
2

No need to use that many variables.

with open(outfilename, 'w') as outfile:
    for fname in filenames:
        with open(fname, 'r') as readfile:
            outfile.write(readfile.read() + "\n\n")
2

I was curious to check more on performance and I used answers of Martijn Pieters and Stephen Miller.

I tried binary and text modes with shutil and without shutil. I tried to merge 270 files.

Text mode -

def using_shutil_text(outfilename):
    with open(outfilename, 'w') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'r') as readfile:
                shutil.copyfileobj(readfile, outfile)

def without_shutil_text(outfilename):
    with open(outfilename, 'w') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'r') as readfile:
                outfile.write(readfile.read())

Binary mode -

def using_shutil_text(outfilename):
    with open(outfilename, 'wb') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'rb') as readfile:
                shutil.copyfileobj(readfile, outfile)

def without_shutil_text(outfilename):
    with open(outfilename, 'wb') as outfile:
        for filename in glob.glob('*.txt'):
            if filename == outfilename:
                # don't want to copy the output into the output
                continue
            with open(filename, 'rb') as readfile:
                outfile.write(readfile.read())

Running times for binary mode -

Shutil - 20.161773920059204
Normal - 17.327500820159912

Running times for text mode -

Shutil - 20.47757601737976
Normal - 13.718038082122803

Looks like in both modes, shutil performs same while text mode is faster than binary.

OS: Mac OS 10.14 Mojave. Macbook Air 2017.

1

The fileinput module provides a natural way to iterate over multiple files

for line in fileinput.input(glob.glob("*.txt")):
    outfile.write(line)
2
  • 1
    This would be even better if it didn't confine itself to reading a line at a time.
    – Marcin
    Jul 19, 2013 at 15:35
  • @Marcin, that is correct. I used to think this was a cool solution - until I saw Martijn Pieter's shutil.copyfileobj humdinger.
    – iruvar
    Jul 19, 2013 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.