11

I have a long-running process which writes a lot of stuff in a file. The result should be everything or nothing, so I'm writing to a temporary file and rename it to the real name at the end. Currently, my code is like this:

filename = 'whatever'
tmpname = 'whatever' + str(time.time())

with open(tmpname, 'wb') as fp:
    fp.write(stuff)
    fp.write(more stuff)

if os.path.exists(filename):
    os.unlink(filename)
os.rename(tmpname, filename)

I'm not happy with that for several reasons:

  • it doesn't clean up properly if an exception occurs
  • it ignores concurrency issues
  • it isn't reusable (I need this in different places in my program)

Any suggestions how to improve my code? Is there a library that can help me out?

1
  • To make it reusable, you can turn it into a context manager. Commented Aug 17, 2012 at 13:48

4 Answers 4

14

You can use Python's tempfile module to give you a temporary file name. It can create a temporary file in a thread safe manner rather than making one up using time.time() which may return the same name if used in multiple threads at the same time.

As suggested in a comment to your question, this can be coupled with the use of a context manager. You can get some ideas of how to implement what you want to do by looking at Python tempfile.py sources.

The following code snippet may do what you want. It uses some of the internals of the objects returned from tempfile.

  • Creation of temporary files is thread safe.
  • Renaming of files upon successful completion is atomic, at least on Linux. There isn't a separate check between os.path.exists() and the os.rename() which could introduce a race condition. For an atomic rename on Linux the source and destinations must be on the same file system which is why this code places the temporary file in the same directory as the destination file.
  • The RenamedTemporaryFile class should behave like a NamedTemporaryFile for most purposes except when it is closed using the context manager, the file is renamed.

Sample:

import tempfile
import os

class RenamedTemporaryFile(object):
    """
    A temporary file object which will be renamed to the specified
    path on exit.
    """
    def __init__(self, final_path, **kwargs):
        tmpfile_dir = kwargs.pop('dir', None)

        # Put temporary file in the same directory as the location for the
        # final file so that an atomic move into place can occur.

        if tmpfile_dir is None:
            tmpfile_dir = os.path.dirname(final_path)

        self.tmpfile = tempfile.NamedTemporaryFile(dir=tmpfile_dir, **kwargs)
        self.final_path = final_path

    def __getattr__(self, attr):
        """
        Delegate attribute access to the underlying temporary file object.
        """
        return getattr(self.tmpfile, attr)

    def __enter__(self):
        self.tmpfile.__enter__()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is None:
            self.tmpfile.delete = False
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)
            os.rename(self.tmpfile.name, self.final_path)
        else:
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)

        return result

You can then use it like this:

with RenamedTemporaryFile('whatever') as f:
    f.write('stuff')

During writing, the contents go to a temporary file, on exit the file is renamed. This code will probably need some tweaks but the general idea should help you get started.

5
  • +1 for NamedTemporaryFile and the context manager. Note: you might loose data without os.fsync() call. See my answer
    – jfs
    Commented Aug 17, 2012 at 20:37
  • @J.F.Sebastian Using os.fsync() may get you closer to a fault tolerant system at the expense of performance, but fsync() still doesn't guarantee that data is on disk, especially for storage with write caches. I'd probably choose to use/not use fsync() depending on the application. blogs.gnome.org/alexl/2009/03/16/ext4-vs-fsync-my-take has some interesting info on this. Commented Aug 18, 2012 at 0:24
  • Philips: the OP specifically asked about fault tolerant, not the fastest way. POSIX allows int fsync(int) { return 0; } but most implementations do something useful. From your link: "Some filesystems may order the metadata writes such that the rename is written to disk, but the contents of the new file are not yet on disk. If we crash at this point this is detected on mount and the file is truncated to 0 bytes. Calling fsync() guarantees that this does not happen. [ext4]".
    – jfs
    Commented Aug 18, 2012 at 3:03
  • Though fsync() might not be necessary for some kernels in the rename() case.
    – jfs
    Commented Aug 18, 2012 at 3:04
  • Thanks, this helps a lot! J.F. is right, I don't care about performance much in this case, so one extra function call is fine.
    – georg
    Commented Aug 18, 2012 at 8:52
6

To write all or nothing to a file reliably:

import os
from contextlib import contextmanager
from tempfile   import NamedTemporaryFile

if not hasattr(os, 'replace'):
    os.replace = os.rename #NOTE: it won't work for existing files on Windows

@contextmanager
def FaultTolerantFile(name):
    dirpath, filename = os.path.split(name)
    # use the same dir for os.rename() to work
    with NamedTemporaryFile(dir=dirpath, prefix=filename, suffix='.tmp') as f:
        yield f
        f.flush()   # libc -> OS
        os.fsync(f) # OS -> disc (note: on OSX it is not enough)
        f.delete = False # don't delete tmp file if `replace()` fails
        f.close()
        os.replace(f.name, name)

See also Is rename() without fsync() safe? (mentioned by @Mihai Stan)

Usage

with FaultTolerantFile('very_important_file') as file:
    file.write('either all ')
    file.write('or nothing is written')

To implement missing os.replace() you could call MoveFileExW(src, dst, MOVEFILE_REPLACE_EXISTING) (via win32file or ctypes modules) on Windows.

In case of multiple threads you could call queue.put(data) from different threads and write to file in a dedicated thread:

 for data in iter(queue.get, None):
     file.write(data)

queue.put(None) breaks the loop.

As an alternative you could use locks (threading, multiprocessing, filelock) to synchronize access:

def write(self, data):
    with self.lock:
        self.file.write(data)
4
  • Thanks, very useful stuff. Some new concepts to learn (trying to wrap my head around how @contextmanager works).
    – georg
    Commented Aug 18, 2012 at 8:55
  • @thg435: the docstring might explain it
    – jfs
    Commented Aug 18, 2012 at 18:35
  • What is the purpose/advantage of the yield statement?
    – Batman
    Commented Dec 30, 2014 at 16:50
  • 1
    @Batman: For the contextmanager decorator. Read the its docs to understand why. And click the link in my previous comment
    – jfs
    Commented Dec 30, 2014 at 16:54
2

The with construct is useful for cleaning up on exit, but not for the commit/rollback system you want. A try/except/else block can be used for that.

You also should use a standard way for creating the temporary file name, for example with the tempfile module.

And remember to fsync before rename

Below is the full modified code:

import time, os, tempfile

def begin_file(filepath):
    (filedir, filename) = os.path.split(filepath)
    tmpfilepath = tempfile.mktemp(prefix=filename+'_', dir=filedir)
    return open(os.path.join(filedir, tmpfilepath), 'wb') 

def commit_file(f):
    tmppath = f.name
    (filedir, tmpname) = os.path.split(tmppath)
    origpath = os.path.join(filedir,tmpname.split('_')[0])

    os.fsync(f.fileno())
    f.close()

    if os.path.exists(origpath):
        os.unlink(origpath)
    os.rename(tmppath, origpath)

def rollback_file(f):
    tmppath = f.name
    f.close()
    os.unlink(tmppath)


fp = begin_file('whatever')
try:
    fp.write('stuff')
except:
    rollback_file(fp)
    raise
else:
    commit_file(fp)
1
1

You could use the lockfile module to lock the file while you are writing to it. Any subsequent attempt to lock it will block until the lock from the previous process/thread has been released.

from lockfile import FileLock
with FileLock(filename):
    #open your file here....

This way, you circumvent your concurrency issues and do not have to clean up any leftover file if an exception occurs.

2
  • +1 the lockfile link. Note: for threads in the same process threading.Lock could be used (or multiprocessing.Lock for processing created via multiprocessing module).
    – jfs
    Commented Aug 17, 2012 at 20:42
  • Thanks, that module appears to be really useful.
    – georg
    Commented Aug 18, 2012 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.