[EDIT: I solved(?) the problem on my own and updated the question accordingly]
I have a Python script which performs daily backups of a large folder (with subfolders). As the folder has become too large and the backup process takes too much time, being such process CPU-bound I needed to modify the script so to make use of the several CPU cores in the server.
NOTE: There probably are better modules and functions to be used in this use-case, but using multiprocessing.Pool()
applied to a subprocess.call
to rsync
is a requirement from which I could not diverge.
- I struggled for a couple of weeks on this because I always ended up having several duplicates in the destination folder (hence the original question); I now have found a clunky but working solution by splitting the file/folder paths and merging them again (using
os.path.join
): is there a better way to do this? - Although the script produces the exact backup of the original folder, I still get some suspicious rsync error messages during the execution and I'm worried that everything will crumble as soon as the wrong combination of processes gets executed: can anyone take a look at those and tell me if I need to worry?
Here is the (minimal working version of the) code I wrote, commented:
#!/usr/bin/env python3
import os
import multiprocessing
import subprocess
#In this test /src/ and /dest/ are in the same folder of the script
source = os.path.join(os.getcwd(),"src")
destination = os.path.join(os.getcwd(),"dest")
def get_pathlist(folder):
pathlist = []
#No need for the FULL path of the dir/file
#The source directory is in a global variable
for root,dirs,files in os.walk(folder):
for f in files:
#Extract the sub-folder (if any)
path = root[len(folder):]
#Extract the filename
item = f
#Store the RELATIVE path in a tuple
pathlist.append((path,item))
for d in dirs:
#Extract the sub-folder (if any)
path = root[len(folder):]
#Extract the folder name
item = d
#Store the RELATIVE path in a tuple
pathlist.append((path,item))
#Return the list of tuples
return pathlist
def backup(path):
#Source = root, path[0] = sub-folder, path[1] = file/dir name
#NB: We input the FULL path of the source file/folder
src = os.path.join(source,path[0],path[1])
#Destination = root, path[0] = sub-folder
#NB: We input the destination folder only (no need for the file/folder name)
dest = os.path.join(destination,path[0])
subprocess.call(['rsync', '-azq', src, dest])
if __name__ == "__main__":
src_pathlist = get_pathlist(source)
with multiprocessing.Pool(len(src_pathlist),maxtasksperchild=1) as mpool:
mpool.map(backup,src_pathlist)
As I mentioned the script produces the intended result but I get some strange errors from rsync. The /src/ structure is as follows (for reference):
/src/
/1.txt
/Folder
/Folder/2.txt
/Folder/New_folder-A
/Folder/New_folder-B
/Folder/New_folder-C
/Folder/New_folder-A/3A.txt
/Folder/New_folder-B/3B.txt
The errors I get are:
rsync: change_dir "/Folder" failed: No such file or directory (2)
rsync: change_dir "/Folder" failed: No such file or directory (2)
rsync: change_dir "/Folder" failed: No such file or directory (2)
rsync: change_dir "/Folder" failed: No such file or directory (2)
rsync: change_dir "/Folder/New_folder-A" failed: No such file or directory (2)
rsync: change_dir#3 "/Folder" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(720) [Receiver=3.1.3]
rsync: change_dir "/Folder/New_folder-B" failed: No such file or directory (2)
rsync: change_dir#3 "/Folder" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(720) [Receiver=3.1.3]
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Can anyone shed some light on this? Any improvement suggestions are VERY welcome.