2

Linux command find with argument exec does a GREAT job executing commands on files/folders regardless whether they contain spaces and special characters. For example:

find . -type f -exec md5sum {} \;

Works great to run md5sum on each file in a directory tree, but executes in a random order. Find does not sort the results, and requires piping to sort to get results in a more human-readable ordering. However, piping to sort eliminates the benefits of exec.

This does not work:

find . -type f | sort | md5sum

Because some filenames contain spaces and special characters.

Also does not work:

find . -type f | sort | sed 's/ /\\ /g' | md5sum

Still does not recognize spaces are part of the filename.

I suppose I can always sort the final result later, but wonder if someone knows an easy way to avoid that extra step by sorting within find?

3
  • The spaces aren't the problem there. md5sum, reading from stdin, doesn't expect a list of filenames, but a single stream to be hashed. Oct 29, 2016 at 18:05
  • That said -- which implementation of find? BSD find can do internal sorting; GNU find cannot (but the GNU toolchain is powerful enough to offer other approaches). Oct 29, 2016 at 18:07
  • BTW, unless you had literal newlines in your filenames, find . -type f | sort | xargs -d $'\n' md5sum would work. I hesitate to make that an answer, because having known bugs (in this case, the literal-newline case) isn't good practice -- even if a bug represents an obscure corner case, sometimes an attacker can make a security hole of such a corner-case. Oct 29, 2016 at 18:12

2 Answers 2

3

With BSD find

A -s argument is available to request lexographic sort order.

find . -s -type f -exec md5sum -- '{}' +

With GNU find

Use NUL delimiters to allow filenames to be processed unambiguously. Assuming you have GNU tools:

find . -type f -print0 | sort -z | xargs -0 md5sum
1
  • Thanks, Charles! Using NUL delimiters is a great idea! I use gnu find, and have no -s Oct 29, 2016 at 18:10
0

Found a working solution

find . -type f -exec md5sum {} + | sort -k 1.33

Sorts the results by comparing the characters starting after the 32 character md5sum result, producing a readable/sorted list.

2
  • 1
    FYI, -exec md5sum {} \; is much slower than -exec md5sum {} +, as it loads a new copy of md5sum for every file to process. Oct 29, 2016 at 18:09
  • Thanks. Replaced my \; with + and edited solution from find . -type f -exec md5sum {} \; | sort -k 1.33 to use a + instead Oct 29, 2016 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.