1

Sorry if this has been answered before but I am having some trouble finding a definitive solution to the following. I have a folder where new files are added daily. Everyday function, fun1, is looped through each file in the folder. What is the best way to ensure that function fun1 runs on each file only once and does not run on the same file again the next day?

e.g. On day 1 Folder has one file: file1. fun1 then runs on file1.
On day 2 a new file, file2, is added to Folder. On day 2 fun1 should run on file2 but skip file1.

fun1 does not actually modify file1, rather it uses the data from file1 to modify an existing file, say file1s.

I thought about approaching this by moving the files that have had fun1 run on them to a different folder with something like

for files = readdir(folder1, join=true)
    fun1(files)
    destination = replace.(file, "folder1"=> "folder2")
    mv.(file, destination)
end

But I don't know if this might introduce problems or complications or if this is frowned upon generally? I could also create a separate set of files that have been "processed" with something like

processed_files = DataFrame(filename=[])
all_files = readdir(folder1, join = true)
for file = setdiff( all_files, processed_files.filename)
    fun1(file)
    push!(processed_files.filename, file)
end

But I don't know if this would be efficient as overtime processed_files would grow very large. I would really appreciate any comments on whether these are sound approaches or if there is a more idiomatic Julia way to achieve this? Thanks so much!

5
  • 1
    I am guessing that with this setup you also read and write the dataframe to disk daily to get the diff? Commented May 8, 2023 at 9:38
  • 2
    An alternative approach could be checking the date and time of creation of each file and comparing it with the date and time of the last time the script was run. Build systems like GNU Make come to mind Commented May 8, 2023 at 9:41
  • 1
    Extending @loonatick's idea, check the timestamp of file1 compared to file1s, if it is newer or file1s is missing, then process file1. Commented May 8, 2023 at 10:47
  • awesome thanks so much guys! I didn't realize I could approach it like this. As the Folder gets larger...say now we grow from file1 to file10^n. Does it still make sense to loop through each file checking time stamps or would it be preferable to just periodically convert Folder to a giant GDF, process each SubDataFrame and then move all processed files to a different folder? I'm just at a loss as to which approach would be more efficient and reliable in the long run? Commented May 9, 2023 at 3:21
  • I could also rename the file after it is processed but would that be more reliable than actually moving the file to a different folder? Sorry I don't really know what I'm doing so just wondering if keeping the folder the same in the directory somehow makes a difference in terms of making things less error prone. Commented May 9, 2023 at 3:42

1 Answer 1

2

If keeping the Julia process alive is an option, one solution would be to monitor the folder for changes using the FileWatcher.watch_folder() function which is part of Julia's standard library.

The documentation can be found here: https://docs.julialang.org/en/v1/stdlib/FileWatching/

Here is an example:

julia> using FileWatching

julia> while true
         filename, event = watch_folder(".")
         @show filename event
       end
filename = "test1"
event = FileWatching.FileEvent(true, false, false)
filename = "test2"
event = FileWatching.FileEvent(true, false, false)

Notice the while true since once a change has occurred, watch_folder is non-blocking and returns the filename and event. In the example above, I created two files named test1 and test2 with touch ..., in the same folder where julia was started.

What's nice about this approach is that no constant polling is done but watch_folder uses the filesystem specific notification system to retrieve events. To my understanding, this is the cleanest solution to your problem.

Don't worry about files which have been added/modified while watch_folder was not "blocking", the changes are queued up, just like in a stream.

Sign up to request clarification or add additional context in comments.

3 Comments

Thanks so much for this amazing answer! Sorry I've been playing around with this and am still really confused about how to run a function on the new files coming in. I am assuming I have to collect filename into an array with something like push(newarray, filename), broadcast the function over newarray and then remove filename with something like setdiff!(newarray, filename) ? But how can I perform these operations on newarray while it is constantly being modified by the while loop? is this a case that would involve @async or Channel?
Sorry a bit slow on the uptake... I think I understand how to implement your solution now. Would it be possible to use watch_folder on multiple folders simultaneously without opening separate instances of Julia?
Glad you worked that out! Watching different folders could be done with threads, that’s fairly easy. You can come over to the Julia discourse if you need more help ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.