I am using Python to write chunks of text to files in a single operation:
open(file, 'w').write(text)
If the script is interrupted so a file write does not complete I want to have no file rather than a partially complete file. Can this be done?
Join Stack Overflow to learn, share knowledge, and build your career.
I am using Python to write chunks of text to files in a single operation:
open(file, 'w').write(text)
If the script is interrupted so a file write does not complete I want to have no file rather than a partially complete file. Can this be done?
Write data to a temporary file and when data has been successfully written, rename the file to the correct destination file e.g
f = open(tmpFile, 'w')
f.write(text)
# make sure that all data is on disk
# see http://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe
f.flush()
os.fsync(f.fileno())
f.close()
os.rename(tmpFile, myFile)
According to doc http://docs.python.org/library/os.html#os.rename
If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file; there may be no way to implement an atomic rename when dst names an existing file
also
The operation may fail on some Unix flavors if src and dst are on different filesystems.
Note:
It may not be atomic operation if src and dest locations are not on same filesystem
os.fsync
step may be skipped if performance/responsiveness is more important than the data integrity in cases like power failure, system crash etc
os.fsync(f)
before f.close()
, as that will ensure the new file's data is actually on disk
– Dan D.
Mar 8 '11 at 11:58
rename
is atomic only within same filesystem on POSIX, so the easiest way is to create tmpFile
in the directory of myFile
.
– darkk
Jan 13 '12 at 15:10
fsync(opendir(filename))
to ensure that rename is written to disk too. This does not affect atomicity of this modification, only relative order of this operation vs prev/next on a different file.
– Dima Tisnek
Mar 14 '14 at 12:09
A simple snippet that implements atomic writing using Python tempfile
.
with open_atomic('test.txt', 'w') as f:
f.write("huzza")
or even reading and writing to and from the same file:
with open('test.txt', 'r') as src:
with open_atomic('test.txt', 'w') as dst:
for line in src:
dst.write(line)
using two simple context managers
import os
import tempfile as tmp
from contextlib import contextmanager
@contextmanager
def tempfile(suffix='', dir=None):
""" Context for temporary file.
Will find a free temporary filename upon entering
and will try to delete the file on leaving, even in case of an exception.
Parameters
----------
suffix : string
optional file suffix
dir : string
optional directory to save temporary file in
"""
tf = tmp.NamedTemporaryFile(delete=False, suffix=suffix, dir=dir)
tf.file.close()
try:
yield tf.name
finally:
try:
os.remove(tf.name)
except OSError as e:
if e.errno == 2:
pass
else:
raise
@contextmanager
def open_atomic(filepath, *args, **kwargs):
""" Open temporary file object that atomically moves to destination upon
exiting.
Allows reading and writing to and from the same filename.
The file will not be moved to destination in case of an exception.
Parameters
----------
filepath : string
the file path to be opened
fsync : bool
whether to force write the file to disk
*args : mixed
Any valid arguments for :code:`open`
**kwargs : mixed
Any valid keyword arguments for :code:`open`
"""
fsync = kwargs.get('fsync', False)
with tempfile(dir=os.path.dirname(os.path.abspath(filepath))) as tmppath:
with open(tmppath, *args, **kwargs) as file:
try:
yield file
finally:
if fsync:
file.flush()
os.fsync(file.fileno())
os.rename(tmppath, filepath)
shutil.move
in case of os.rename
failing. This allows it to work across FS boundaries.
– Nils Werner
Jul 18 '16 at 10:16
shutils.move()
was non-atomic due to shutils.copy2()
and shutils.remove()
called in succession. The new implementation (see edit) will now instead create the file in the current directory and also handle exceptions better.
– Nils Werner
Jul 19 '16 at 12:23
open('test.txt', 'r') as src:
is used to read the file contents. Writing in this sense is atomic but reading might not be the same case. For file types like .ini
playup with decorators when used with configparser for read operations. Not sure this sample completely justifies the atomicity around reading from same file over 200000 threads. This will throw Too Many Open Files error.
– bh4r4th
Jan 10 '20 at 5:27
Since it is very easy to mess up with the details, I recommend using a tiny library for that. The advantage of a library is that it takes care all these nitty-gritty details, and is being reviewed and improved by a community.
One such library is python-atomicwrites
by untitaker which even has proper Windows support:
From the README:
from atomicwrites import atomic_write
with atomic_write('foo.txt', overwrite=True) as f:
f.write('Hello world.')
# "foo.txt" doesn't exist yet.
# Now it does.
I’m using this code to atomically replace/write a file:
import os
from contextlib import contextmanager
@contextmanager
def atomic_write(filepath, binary=False, fsync=False):
""" Writeable file object that atomically updates a file (using a temporary file).
:param filepath: the file path to be opened
:param binary: whether to open the file in a binary mode instead of textual
:param fsync: whether to force write the file to disk
"""
tmppath = filepath + '~'
while os.path.isfile(tmppath):
tmppath += '~'
try:
with open(tmppath, 'wb' if binary else 'w') as file:
yield file
if fsync:
file.flush()
os.fsync(file.fileno())
os.rename(tmppath, filepath)
finally:
try:
os.remove(tmppath)
except (IOError, OSError):
pass
Usage:
with atomic_write('path/to/file') as f:
f.write("allons-y!\n")
It’s based on this recipe.
Answers on this page are quite old, there are now libraries that do this for you.
In particular safer
is a library designed to help prevent programmer error from corrupting files, socket connections, or generalized streams. It's quite flexible and amongst other things it has the option to use either memory or temporary files, you can even keep the temp files in case of failure.
Their example is just what you want:
# dangerous
with open(filename, 'w') as fp:
json.dump(data, fp)
# If an exception is raised, the file is empty or partly written
# safer
with safer.open(filename, 'w') as fp:
json.dump(data, fp)
# If an exception is raised, the file is unchanged.
It's in PyPI, just install it using pip install --user safer
or get the latest at https://github.com/rec/safer
Atomic solution for Windows to loop folder and rename files. Tested, atomic to automate, you can increase probability to minimize risk not to event of having same file name. You random library for letter symbols combinations use random.choice method, for digit str(random.random.range(50,999999999,2). You can vary digits range as you want.
import os import random
path = "C:\\Users\\ANTRAS\\Desktop\\NUOTRAUKA\\"
def renamefiles():
files = os.listdir(path)
i = 1
for file in files:
os.rename(os.path.join(path, file), os.path.join(path,
random.choice('ABCDEFGHIJKL') + str(i) + str(random.randrange(31,9999999,2)) + '.jpg'))
i = i+1
for x in range(30):
renamefiles()