5

For a media library where I'd like to update entries when they change in the file system I wanted to give the 'new' java.nio file watching features a try by using this example.
I was expecting to get useful events when a file is being created, moved (renamed) or deleted, but here is what's happening when watching folders on windows7 (haven't tried the other operating systems yet):

Format:
[ThreadName] DEBUG 2012-04-09 18:20:35.934 GroupNumber-COMMAND: Path
ThreadName: Each watch folder runs in its own thread having a distinct id
GroupNumber: Messages sent with the same GroupNumber are being sent at the same time (usually..)
COMMAND: The received command
Path: The received path

Rename:
[Watch0] DEBUG 2012-04-09 18:20:35.934 2-ENTRY_DELETE: C:\tmp\tmp\test.avi
[Watch0] DEBUG 2012-04-09 18:20:35.935 2-ENTRY_CREATE: C:\tmp\tmp\test1.avi
[Watch0] DEBUG 2012-04-09 18:20:35.936 3-ENTRY_MODIFY: C:\tmp\tmp
[Watch0] DEBUG 2012-04-09 18:20:35.937 4-ENTRY_MODIFY: C:\tmp\tmp\test1.avi

[Watch4] DEBUG 2012-04-09 18:43:47.965 18-ENTRY_DELETE: F:\tmp\test.avi
[Watch4] DEBUG 2012-04-09 18:43:47.966 18-ENTRY_CREATE: F:\tmp\test1.avi
[Watch4] DEBUG 2012-04-09 18:43:47.967 19-ENTRY_MODIFY: F:\tmp\test1.avi

Create:
[Watch0] DEBUG 2012-04-09 18:22:02.055 5-ENTRY_CREATE: C:\tmp\test.avi
[Watch0] DEBUG 2012-04-09 18:22:02.066 6-ENTRY_MODIFY: C:\tmp\test.avi
[Watch0] DEBUG 2012-04-09 18:22:03.460 7-ENTRY_MODIFY: C:\tmp\test.avi
//Note the 1.4'' delay between the last two messages. 
//This is the time required to actually copy the file

Move in same watch folder:
[Watch0] DEBUG 2012-04-09 18:18:42.395 0-ENTRY_DELETE: C:\tmp\test.avi
[Watch0] DEBUG 2012-04-09 18:18:42.396 0-ENTRY_MODIFY: C:\tmp\tmp
[Watch0] DEBUG 2012-04-09 18:18:42.396 1-ENTRY_CREATE: C:\tmp\tmp\test.avi
[Watch0] DEBUG 2012-04-09 18:18:42.396 1-ENTRY_MODIFY: C:\tmp\tmp\test.avi

Move to other watch folder on same drive:
[Watch1] DEBUG 2012-04-09 18:23:24.341 8-ENTRY_CREATE: C:\tmp2\test.avi
[Watch0] DEBUG 2012-04-09 18:23:24.341 8-ENTRY_DELETE: C:\tmp\test.avi
[Watch1] DEBUG 2012-04-09 18:23:24.342 10-ENTRY_MODIFY: C:\tmp2\test.avi
//The two 8 are lying. Both messages are being sent from different threads
//and the shared counter hasn't been incremented by any yet. The next entry has been 
//incremented by two!

Move to other watch folder on different drive:
[Watch4] DEBUG 2012-04-09 18:25:42.324 11-ENTRY_CREATE: F:\tmp\test.avi
[Watch4] DEBUG 2012-04-09 18:25:42.338 12-ENTRY_MODIFY: F:\tmp\test.avi
[Watch4] DEBUG 2012-04-09 18:25:42.703 13-ENTRY_MODIFY: F:\tmp\test.avi
[Watch3] DEBUG 2012-04-09 18:25:49.433 14-ENTRY_DELETE: C:\tmp2\test.avi
//Note that the last delete message is being sent from another thread then the first ones.
//This is because the source and destination WatchDirs aren't the same

Delete:
[Watch9] DEBUG 2012-04-05 21:22:02.921 ENTRY_DELETE: C:\tmp\test (2011).mkv

Instead of having a single event, there are a set of 'command + path' that have to be interpreted. E.g. a delete consists of a single command, whereas rename and 'move in same folder' start with a delete command as well but will be defined by their future commands. Additionally, multiple files could be e.g. moved in parallel, which will end up in a random list of commands belonging to different operations having to be sorted somehow.

The best I could come up with is this class, where events are being enqueued when being received and then checked a moment (1s) after having been received in another thread (to give some time if other events are being generated and belong to the same 'event group').

This works if renaming, moving, creating or deleting a single file, but nothing will work anymore if starting to copy multiple files in parallel or copy a batch of files.

Does any implementation of what I need already exist (seems like a common use case)? Or has someone a good idea how to approach this problem to cover all cases?

In the end, it will have to work for windows, linux and osx.

A more complex example which should be supported as well

[Watch0] DEBUG 2012-04-09 19:10:17.774 0-ENTRY_CREATE: C:\tmp\tmp\testlarge.avi
[Watch0] DEBUG 2012-04-09 19:10:17.825 0-ENTRY_MODIFY: C:\tmp\tmp\testlarge.avi
[Watch0] DEBUG 2012-04-09 19:10:17.826 1-ENTRY_MODIFY: C:\tmp\tmp
[Watch0] DEBUG 2012-04-09 19:12:09.516 2-ENTRY_DELETE: C:\tmp\tmp\testsmall.avi
[Watch0] DEBUG 2012-04-09 19:12:09.516 3-ENTRY_CREATE: C:\tmp\testsmall.avi
[Watch0] DEBUG 2012-04-09 19:12:09.517 3-ENTRY_MODIFY: C:\tmp\testsmall.avi
[Watch0] DEBUG 2012-04-09 19:12:09.521 4-ENTRY_MODIFY: C:\tmp\tmp
[Watch0] DEBUG 2012-04-09 19:14:13.025 5-ENTRY_MODIFY: C:\tmp\tmp\testlarge.avi

Here, a small file is being moved while a large file is being created.

3 Answers 3

4
+50

The events you are getting are the events you are going to get. On Windows. On Linux, when you watch a folder, you're only going to get events for that folder, not for files in that folder. I believe it is even gnarlier on OS X, though I've never dealt with it there. This means if you're watching /tmp you will see ENTRY_CREATE /tmp/tmp/ but not ENTRY_CREATE /tmp/tmp/test.avi

I handled a similar problem by watching the folder, accumulating change notifications in a Set, then periodically operating on all the files mentioned in the Set. (There was one Set being updated and a different one being scanned and cleared, and I'd swap which Set was which.)

In short, I gave up on trying to figure out when files were moved or copied or appended to. I simply treated every create or modify as creating a new file unrelated to previous files. Anything more was just unworkable in practice, at least in my case, where I was tracking millions of files on a server.

5
  • Thanks for the input. It looks like oracle forgot to implement a layer... Before starting, I was expecting to add a file change listener and to be notified like this without having to implement the message handling myself. Maybe they had come to the same conclusion: 'Anything more was just unworkable in practice'. This functionality now seems utterly useless to me.
    – Philippe
    Apr 19, 2012 at 8:17
  • @Philippe, I found the functionality invaluable for my situation. I believe one of the big motivators for its creation was the ability to live-update UI windows showing lists of files in a folder as those file lists changed via external actions. You just have to accept that for this to never miss an event, it has to be implemented at a very low level, and thus has to be extremely simple and efficient to avoid slowing down the whole file system. You never said what you wanted to do with this file change information, so I can't help you more. For me it was enough just to trigger a rescan.
    – Old Pro
    Apr 19, 2012 at 19:45
  • my application manages video, audio and image files to be streamed through DLNA. Currently, a scan button has to be pressed to add new files. I'd like to change it to watch configured folders and automatically trigger an action upon a file change, when the file has been fully written to disk. Create: scan the new file and insert the data into the library, move/rename: update the path of the file in the media library, delete: delete the data from the media library. I can't think of a way to cover all cases, especially differentiating create and move to watch folder on different drive.
    – Philippe
    Apr 20, 2012 at 6:14
  • 1
    @Philippe, just give up on move/rename/modify as a separate thing and treat then as delete followed by create. If you have a way of fingerprinting the files you can use that to see if a newly created file is the same as one recently deleted and then copy over the metadata if you really need to preserve metadata not stored in the file (like ratings or play count).
    – Old Pro
    Apr 20, 2012 at 6:58
  • I have to give this a thorough thought, but working with a fingerprint (hash) could solve the issues. Part of the properties come from the file itself (e.g. duration, resolution for videos) and other bits can be entered manually or gathered with plugins; the latter will have to be preserved and the date inserted property shouldn't be modified either. If you'd like to have more details about the DB and condition framework, it's described in this question stackoverflow.com/questions/8301418/… Thanks a lot for your help!
    – Philippe
    Apr 20, 2012 at 8:56
3

Not very familiar with java nio file watcher, but I used jnotify library to get events from the file system, it was very good and was working with windows and linux.

If you want to use nio any way try to take a look here: https://blogs.oracle.com/thejavatutorials/entry/watching_a_directory_for_changes

1
  • Thought I'd use the functions now contained in the framework, but as jnotify seems to do exactly what I want, it might be better
    – Philippe
    Apr 8, 2012 at 15:29
3

One thing I've seen is that you create a new Thread for each event and you use na lock object to synchronize your Threads.

Better would be if you create an ExecutorService i.e. rewrite your code like:

private ExecutorService executorService;


private class WatchEventHandler implements Callable {

    WatchEvent<?> event;

    public WatchEventHandler(final WatchEvent<?> event) {
        this.event = event;
    }

    @Override
    public void call() throws Exception{
        // do something with the event

        // fireFileWatchAction();


    }    
}


public ManagedFolderWatcher(DOManagedFile managedFolder) throws IOException {
   executorService = Executors.newFixedThreadPool(10);
   ...
}

and in your run/process events method

public void run() {
    try {
       while(true) {

          WatchKey key = watcher.take();
          ....

          for (WatchEvent<?> event : key.pollEvents()) {
              ...
              // in my oppinon there is no need to delay the event 
              executorService.schedule(new WatchEventHandler(event));
          }
          ...

       }
    } catch (InterruptedException ie) {
        // todo: propper error handling
        log.error("Thread interrupted", ie);
    } catch (ClosedWatchServiceException cwse) {
        // todo: propper error handling
        log.error("WatchService allready closed.", cwse);
    }

} 

When you implement it in this way you don't even need your lock object eventEntrySyncObj and your Callable/Command has all the informations needed to fire the fileWatchAction event.

5
  • Looks clean. If you don't delay the event, when do you fire a delete? It could be the start of a rename or move.
    – Philippe
    Apr 8, 2012 at 19:54
  • Currently you create an FileWatchAction for each Event
    – andih
    Apr 9, 2012 at 5:36
  • 2
    If you do something like int count = 0; for (WatchEvent<?> event : key.pollEvents()) {++count; ...} log.debug("Handled {} events", count); You'll see that a mv / rename creates two events for one key belonging together. So instead of passing each Event to the Watchhandler you could pass the Key to the WatchHandler and let the WatchHandler poll the events belonging together. Another optimization would be to use more than one MonitorThread / ManagedFolderWatcher.
    – andih
    Apr 9, 2012 at 5:47
  • I hadn't noticed keys can be sent in groups! I've updated the log messages in the first post to include the group number. Besides delete, rename and move in same watch folder, the operations aren't trivial to handle.
    – Philippe
    Apr 9, 2012 at 17:41
  • @andih I wanted to do similar thing, I wanted to react on rename and move event. Rename seems always to be ENTRY_DELETE and ENTRY_CREATE in the same group. I have problem with handling move event. It seems to be ENTRY_DELETE and ENTRY_MODIFY, but sometimes they are not in the same group (Win7). Can I even make such assumptions on groups - that ENTRY_DELETE and ENTRY_CREATE on the same group (in this order) in the same group always means renaming?
    – Damian
    Apr 17, 2012 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.