16

I have a caching engine on the server which caches all files accessed under a root directory. I'm thinking of using Chokidar to watch the entire directory tree (recursively) for file changes and update the cache accordingly. But I'm concerned about what would happen if a sub directory contained hundreds of thousands of files - How efficient would Chokidar be?

3
  • Can't you rely on your OS's built-in buffer cache to handle the caching of frequently accessed files?
    – robertklep
    Commented Oct 13, 2013 at 10:25
  • Actually this escaped my mind... Nevertheless, I still need a server cache because quite a few of the files get preprocessed before being served (for example all .js files are automatically minified) and then they are all gziped. If I rely on the buffer cache, then I'd still waste a lot of processing doing the minification and gzipping every time instead of serving the gzipped content directly from memory.
    – Jon
    Commented Nov 14, 2013 at 23:31
  • Perhaps the preprocessing step in your app could store the result in some form of cache? Watching 100K files is probably not going to work well.
    – robertklep
    Commented Nov 15, 2013 at 6:57

1 Answer 1

18

Chokidar's efficiency depends on which operating system it's running on.

On OS X, it uses a module that provides access to the native fsevents API, which is extremely efficient.

On other systems, it uses node.js's fs.watch or fs.watchFile APIs. Under the hood, fs.watch uses various system APIs to be notified of changes, which may be reasonably efficient. fs.watchFile uses stat polling, which would definitely be inappropriate for directories as large as you're describing.

My suggestion is that you set the chokidar option usePolling: false and give it a try, monitoring your cpu load.

Update (July 2015): Chokidar has been substantially improved since this was originally written, and polling is no longer the default on any platform.

2
  • 1
    How things going in Oct 2017? Did chokidar became faster on Windows / Linux? Are there any tests with chokidar performance? How deeply nested directories chokidar is good for? I'm planning to watch 10,000 of files with it. But not sure if its good way to go. Commented Oct 5, 2017 at 18:00
  • 7
    @SystemsRebooter I'm using it to monitor 50k+ files on windows and it works fine. After 100k it uses quite a lot of memory and fails to see additions
    – the-noob
    Commented Oct 27, 2017 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.