I have a large set of files located across a series of directories on a windows 2003 server. There are upwards of a million files in each directory. The Windows server uses iSCSI to connect to an Equalogics SAN.
I have an application that needs to determine if a set of files exists - the application needs to check for the existence of up to a million files per directory.
I have tried a variety of techniques / scripting languages including perl, vbscript, dos batch files and I can not obtain greater than about 250 files checks per second. This works out to almost 50 minutes to check for 800,000 files. I tried multithreading a perl program to check for multiple files at a time, but this did not help.
I have also tried to list all of the files in the directory using dir, ls, find (via cygwin), and it takes many minutes for it to start outputting any file names at all. This isn't a great approach anyway, because there are more files than I actually need to check for.
Is there a way I can force windows to do a "read ahead" on the directory, and get the files into a cache?
Is there a better way to approach this find of a problem?