Sorry if this has been answered before but I am having some trouble finding a definitive solution to the following. I have a folder where new files are added daily. Everyday function, fun1, is looped through each file in the folder. What is the best way to ensure that function fun1 runs on each file only once and does not run on the same file again the next day?
e.g. On day 1 Folder has one file: file1. fun1 then runs on file1.
On day 2 a new file, file2, is added to Folder. On day 2 fun1 should run on file2 but skip file1.
fun1 does not actually modify file1, rather it uses the data from file1 to modify an existing file, say file1s.
I thought about approaching this by moving the files that have had fun1 run on them to a different folder with something like
for files = readdir(folder1, join=true)
fun1(files)
destination = replace.(file, "folder1"=> "folder2")
mv.(file, destination)
end
But I don't know if this might introduce problems or complications or if this is frowned upon generally? I could also create a separate set of files that have been "processed" with something like
processed_files = DataFrame(filename=[])
all_files = readdir(folder1, join = true)
for file = setdiff( all_files, processed_files.filename)
fun1(file)
push!(processed_files.filename, file)
end
But I don't know if this would be efficient as overtime processed_files would grow very large. I would really appreciate any comments on whether these are sound approaches or if there is a more idiomatic Julia way to achieve this? Thanks so much!
file1compared tofile1s, if it is newer orfile1sis missing, then processfile1.Foldergets larger...say now we grow fromfile1tofile10^n. Does it still make sense to loop through each file checking time stamps or would it be preferable to just periodically convertFolderto a giant GDF, process each SubDataFrame and then move all processed files to a different folder? I'm just at a loss as to which approach would be more efficient and reliable in the long run?