9

I'm running a script that looks at all the files in a directory and its subdirectories.

The script has been running for a day, and I'd like to estimate how long it will keep running. I know how many files it processed so far (73,000,000), but I don't know the total number of files.

What is the fastest way to count the files?

I tried right-clicking on the directory and selecting "properties", and it's slowly counting up. I tried redirecting ls into a file, and it's just churning & churning...

Should I write a program in c?

1

4 Answers 4

7

The simplest way:

find <dir> -type f | wc -l

Slightly faster, perhaps:

find <dir> -type f -printf '\n' | wc -l
2
  • 1
    do you think find will be faster than ls? Jun 4, 2015 at 19:32
  • It'll be faster than saving ls's output to a file. It'll be on par with ls -R | wc -l. Jun 4, 2015 at 20:02
6

I did a quick research. Using a directory with 100,000 files I compared the following commands:

ls -R <dir>
ls -lR <dir>
find <dir> -type f

I ran them twice, once redirecting into a file (>file), and once piping into wc (|wc -l). Here are the run times in seconds:

        >file   |wc
ls -R     14     14
find      89     56
ls -lR    91     82

The difference between >file and |wc -l is smaller than the difference between ls and find.

It appears that ls -R is at least 4x faster than find.

1
  • 1
    I can't undo my upvote, but this is wrong. ls -R may be faster, but it also gives incorrect results. First off, this counts directories as well as files. Worse yet, the output is formatted, and those formatting lines are counted as well. Jul 13, 2018 at 18:23
2

Fastest I know about:

ls | wc -l

Note: keep in mind though that it lists all nodes inside a directory, including subdirectories and the two references to the current and the parent directory (. & ..).

If you need the recursive count of files in all subdirectories (as opposed to everything including subdirectories inside the current directory), then you can add the "recursive" flag to the ls command:

ls -R | wc -l

If you compare this in speed to the suggestion using find you will see that it is much faster (factor 2 to 10), but keep in mind the note above.

7
  • This misses the files in subdirectories. Jun 4, 2015 at 19:39
  • @RSamuelKlatchko Yeah, just added that option as a seconf alternative. The OPs question is a bit vague about this...
    – arkascha
    Jun 4, 2015 at 19:41
  • I tried ls -R, and I'm redirecting it into a file instead of piping into wc, so that I can see how far it got. It's still running. So I was wondering if there's a fastest way. Jun 4, 2015 at 19:50
  • Hm, ls is pretty fast. Could it be that your performance issue is writing to the file? Keep in mind that writing files is a very slow process...
    – arkascha
    Jun 4, 2015 at 19:51
  • Sure, I see the problem, but slowing it down further certainly won't help :-) I doubt you can find or implement something really faster than the ls command.
    – arkascha
    Jun 4, 2015 at 20:20
0

ls is not fast at all, and for your purpose is not even valuable: indeed ls prints an alhabetically sorted list of items, so have to wait for the OS to return the whole list of entries, sort them, print in the standard output, an then filter the result looking for newline characters.

A looooot of work for a simple task and even worse: if some of your file has a newline in the name, you'll count it more than once.

find, on the other hand, dosn't sort. It also have the advantage of immediatly executing the actions when the buffer is returned from the File System, so you'll start seeing result immediatly, and will consume far less memory.

So prefer this approach instead:

find . -mindepth 0 -maxdepth 0 -ignore_readdir_race -prinf x | wc -m

It will print an "x" in the standard output for every item found in the current directory (excluded the current directory itself, with -mindepth 1), and don't recurse (-maxdepth 1), then count the chracters.

Given that the folder is very full, -ignore_readdir_race will ignore errors for files deleted while counting

If you want to know the current count, redirect the output to a file (possibly in a tmpfs, so all is in-memory, and you won't produce a bottleneck), then detach the process. When you want to know the current counter's value, simply wc -m /tmp/count.txt:

nohup find . -mindepth 1 -maxdepth 1 -ignore_readdir_race -printf x > /tmp/count.txt &

Then when you want to see the actual count:

wc -m /tmp/count.txt

Or just keep watching it increase...

watch wc -m /tmp/count.txt

Have fun

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.