I have a caching engine on the server which caches all files accessed under a root directory. I'm thinking of using Chokidar to watch the entire directory tree (recursively) for file changes and update the cache accordingly. But I'm concerned about what would happen if a sub directory contained hundreds of thousands of files - How efficient would Chokidar be?
-
Can't you rely on your OS's built-in buffer cache to handle the caching of frequently accessed files?– robertklepCommented Oct 13, 2013 at 10:25
-
Actually this escaped my mind... Nevertheless, I still need a server cache because quite a few of the files get preprocessed before being served (for example all .js files are automatically minified) and then they are all gziped. If I rely on the buffer cache, then I'd still waste a lot of processing doing the minification and gzipping every time instead of serving the gzipped content directly from memory.– JonCommented Nov 14, 2013 at 23:31
-
Perhaps the preprocessing step in your app could store the result in some form of cache? Watching 100K files is probably not going to work well.– robertklepCommented Nov 15, 2013 at 6:57
1 Answer
Chokidar's efficiency depends on which operating system it's running on.
On OS X, it uses a module that provides access to the native fsevents
API, which is extremely efficient.
On other systems, it uses node.js's fs.watch
or fs.watchFile
APIs. Under the hood, fs.watch
uses various system APIs to be notified of changes, which may be reasonably efficient. fs.watchFile
uses stat polling, which would definitely be inappropriate for directories as large as you're describing.
My suggestion is that you set the chokidar option usePolling: false
and give it a try, monitoring your cpu load.
Update (July 2015): Chokidar has been substantially improved since this was originally written, and polling is no longer the default on any platform.
-
1How things going in Oct 2017? Did chokidar became faster on Windows / Linux? Are there any tests with chokidar performance? How deeply nested directories chokidar is good for? I'm planning to watch 10,000 of files with it. But not sure if its good way to go. Commented Oct 5, 2017 at 18:00
-
7@SystemsRebooter I'm using it to monitor 50k+ files on windows and it works fine. After 100k it uses quite a lot of memory and fails to see additions– the-noobCommented Oct 27, 2017 at 14:23