7

I'm trying to calculate directory sizes in a way that divides the load so that the user can see counting progress. I thought a logical way to do this would be to first create the directory tree then do an operation counting the length of all the files.

The thing that comes to me as unexpected is that the bulk of time (disk I/O) comes from creating the directory tree, then going over the FileInfo[] comes nearly instantly with virtually no disk I/O.

I've tried with both Directory.GetDirectories(), simply creating a tree of strings of the directory names, and using a DirectoryInfo object, and both methods still take the bulk of the I/O time (reading the MFT of course) compared to going over all the FileInfo.Length for the files in each directory.

I guess there's no way to reduce the I/O to make the tree significantly, I guess I'm just wondering why this operation takes significantly more time compared to going over the more numerous files?

Also, if anyone could recommend a non-recursive way to tally things up (since it seems I need to just split up the enumeration and balance it in order to make the size tallying more responsive). Making a thread for each subdirectory off the base and letting scheduler competition balance things out would probably not be very good, would it?

EDIT: Repository for this code

8
  • I've also struggled with calculating directory size. I've done exactly what you've done. Tried >fileInfo[] and then >Directory.GetDirectories(). But I still don't know of any better way. Jun 26, 2012 at 17:57
  • You're saying that calling GetDirectories() takes a long time? I have not seen that but then again, I've never done this with a large amount of directories. Also, why would you care if its recursive? This is a recursive task and you're never going to have so many nested directories that you'll blow stack. Jun 26, 2012 at 18:00
  • refer stackoverflow.com/questions/468119/… Jun 26, 2012 at 18:00
  • Side note: beware of hard/soft links and mount points, this could make results wrong. Also have some sort of protection against infinite recursion just in case. Jun 26, 2012 at 18:14
  • Please see "Stack Overflow does not allow tags in titles". Jun 26, 2012 at 18:47

1 Answer 1

4

You can utilize Parallel.ForEach to run the directory size calculation in parallel fashion. You can get the GetDirectories and run the Parallel.ForEach on each node. You can use a variable to keep track of size and display that to the user. Each parallel calculation would be incrementing on the same variable. If needed use lock() to synchronize between parallel executions.

4
  • You should code it so that only un-related directories are parallelized and there will be no reason to lock beyond that. Though with most disks I'm not sure what parallelizing will gain you. Disk IOs seem synchronous in nature. All that you can really make parallel is the actual addition of totals which should be negligible Jun 26, 2012 at 18:02
  • @JasonMalinowski Really...I had no idea. Do most OSs know how to take advantage of that? I knew it was much faster due to the no moving parts bit, had no idea it enabled parallel too. Jun 26, 2012 at 18:08
  • Good idea, I never heard of Parallel.Foreach(), sounds very useful.
    – j.i.h.
    Jun 26, 2012 at 20:39
  • @GeorgeMauer Even spinning disks supported "native command queuing" which would allow for more than one IO to be sent to the drive so it can choose how to best satisfy the requests. Also consider situations where the filesystem had already cached part of the directory structure (so enumerating is CPU bound) and other parts weren't. I'm not saying that you'll see magic performance speedups due to parallelization, but there are many reasons why such a speedup could happen. Jun 27, 2012 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.