325

I'm looking to run a Linux command that will recursively compare two directories and output only the file names of what is different. This includes anything that is present in one directory and not the other or vice versa, and text differences.

0

7 Answers 7

485

From the diff man page:

-q   Report only whether the files differ, not the details of the differences.
-r   When comparing directories, recursively compare any subdirectories found.

Example command:

diff -qr dir1 dir2

Example output (depends on locale):

$ ls dir1 dir2
dir1:
same-file  different  only-1

dir2:
same-file  different  only-2
$ diff -qr dir1 dir2
Files dir1/different and dir2/different differ
Only in dir1: only-1
Only in dir2: only-2
4
  • 1
    Thanks - the diff man page in CentOS 7 describes -q as "report only when files differ", which is less clear than what you wrote.
    – Chap
    Commented Dec 12, 2018 at 17:58
  • 5
    This compares the actual content of the files which is often what one wants, however, the rsync answer gives the option to only look at the file names and sizes, and not the content; that is sometimes desirable.
    – steveb
    Commented Feb 13, 2019 at 21:18
  • Works on macOS, too. Commented Aug 14, 2019 at 10:35
  • Also, can include -x PATTERN in command to exclude certain subdirectories. For example, diff -qr repo1 repo2 -x ".git" will compare two directories but will exclude file paths with ".git" in them.
    – ViFI
    Commented Apr 16, 2020 at 19:12
46

You can also use rsync

rsync -rv --size-only --dry-run /my/source/ /my/dest/ > diff.out
5
  • 9
    --size-only will miss files of identical size but different content, e.g. old/version.txt "29a" new/version.txt "29b". Use instead: rsync -ric --dry-run old/ new/ where the "-i" argument allows to obtain the file list directly via rsync -ric --dry-run old/ new/ | cut -d" " -f 2
    – iolsmit
    Commented Mar 24, 2015 at 17:15
  • 14
    This is great if you are only looking for missing files (especially across network shares), because it doesn't compare the contents. This helped me find a few files that failed when migrating to a new NAS. Commented Dec 26, 2015 at 18:26
  • 7
    Be sure to include the trailing slash for paths specified on rsync's command line. WIthout them, this won't work correctly, and rsync will likely just enumerate all file names! Commented Nov 18, 2017 at 0:37
  • 2
    With regard to the comments on not checking the content. This is sometimes desirable, at least as first pass.
    – steveb
    Commented Feb 13, 2019 at 21:14
  • @OverZealous Yes, but the odds are good that the size won't be the same to the byte. Still, I would just exclude the --size-only to check full contents.
    – geneorama
    Commented Sep 9, 2020 at 21:43
18

If you want to get a list of files that are only in one directory and not their sub directories and only their file names:

diff -q /dir1 /dir2 | grep /dir1 | grep -E "^Only in*" | sed -n 's/[^:]*: //p'

If you want to recursively list all the files and directories that are different with their full paths:

diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}'

This way you can apply different commands to all the files.

For example I could remove all the files and directories that are in dir1 but not dir2:

diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}' xargs -I {} rm -r {}
13

The approach of running diff -qr old/ new/ has one major drawback: it may miss files in newly created directories. E.g. in the example below the file data/pages/playground/playground.txt is not in the output of diff -qr old/ new/ whereas the directory data/pages/playground/ is (search for playground.txt in your browser to quickly compare). I also posted the following solution on Unix & Linux Stack Exchange, but I'll copy it here as well:

To create a list of new or modified files programmatically the best solution I could come up with is using rsync, sort, and uniq:

(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq

Let me explain with this example: we want to compare two dokuwiki releases to see which files were changed and which ones were newly created.

We fetch the tars with wget and extract them into the directories old/ and new/:

wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29d.tgz
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29.tgz
mkdir old && tar xzf dokuwiki-2014-09-29.tgz -C old --strip-components=1
mkdir new && tar xzf dokuwiki-2014-09-29d.tgz -C new --strip-components=1

Running rsync one way might miss newly created files as the comparison of rsync and diff shows here:

rsync -rcn --out-format="%n" old/ new/

yields the following output:

VERSION
doku.php
conf/mime.conf
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php

Running rsync only in one direction misses the newly created files and the other way round would miss deleted files, compare the output of diff:

diff -qr old/ new/

yields the following output:

Files old/VERSION and new/VERSION differ
Files old/conf/mime.conf and new/conf/mime.conf differ
Only in new/data/pages: playground
Files old/doku.php and new/doku.php differ
Files old/inc/auth.php and new/inc/auth.php differ
Files old/inc/lang/no/lang.php and new/inc/lang/no/lang.php differ
Files old/lib/plugins/acl/remote.php and new/lib/plugins/acl/remote.php differ
Files old/lib/plugins/authplain/auth.php and new/lib/plugins/authplain/auth.php differ
Files old/lib/plugins/usermanager/admin.php and new/lib/plugins/usermanager/admin.php differ

Running rsync both ways and sorting the output to remove duplicates reveals that the directory data/pages/playground/ and the file data/pages/playground/playground.txt were missed initially:

(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq

yields the following output:

VERSION
conf/mime.conf
data/pages/playground/
data/pages/playground/playground.txt
doku.php
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php

rsync is run with theses arguments:

  • -r to "recurse into directories",
  • -c to also compare files of identical size and only "skip based on checksum, not mod-time & size",
  • -n to "perform a trial run with no changes made", and
  • --out-format="%n" to "output updates using the specified FORMAT", which is "%n" here for the file name only

The output (list of files) of rsync in both directions is combined and sorted using sort, and this sorted list is then condensed by removing all duplicates with uniq

2
  • Couldn't you just run it backwards (diff new/ old/) to see which directories were deleted?
    – Jacques
    Commented Apr 5, 2017 at 8:24
  • Running diff -qr new/ old/ on the example above with the dokuwiki tars produces the same output as diff -qr old/ new/ – i.e. you see that the directory is new/missing but not the files in it
    – iolsmit
    Commented Apr 5, 2017 at 15:08
12

On my linux system to get just the filenames

diff -q /dir1 /dir2|cut -f2 -d' '
4
  • 7
    I don't put filenames with spaces on my linux system. ;)
    – gerardw
    Commented Dec 4, 2014 at 20:28
  • 6
    I did not mean to impute this to you... ;-p Just as a hint for somebody who does...
    – michuelnik
    Commented Dec 5, 2014 at 16:34
  • does not work for me. My directory structure is like below audit-0.0.234/audit-data-warehouse-0.0.234/ audit-0.0.235/audit-data-warehouse-0.0.235/ Commented Aug 4, 2017 at 9:57
  • 1
    diff -qrN /dir1 /dir2 | cut -f2 -d' ' works fine for me!
    – Francesco
    Commented Dec 4, 2019 at 8:52
2

I have a directory.

$ tree dir1
dir1
├── a
│   └── 1.txt
├── b
│   └── 2.txt
└── c
    ├── 3.txt
    ├── 4.txt
    └── d
        └── 5.txt

4 directories, 5 files

I have another directory.

$ tree dir2
dir2
├── a
│   └── 1.txt
├── b
└── c
    ├── 3.txt
    ├── 5.txt
    └── d
        └── 5.txt

4 directories, 4 files

I can diff two directories.

$ diff <(cd dir1; find . -type f | sort) <(cd dir2; find . -type f| sort)
--- /dev/fd/11  2022-01-21 20:27:15.000000000 +0900
+++ /dev/fd/12  2022-01-21 20:27:15.000000000 +0900
@@ -1,5 +1,4 @@
 ./a/1.txt
-./b/2.txt
 ./c/3.txt
-./c/4.txt
+./c/5.txt
 ./c/d/5.txt
2
  • 2
    This approach is nice, but only lists files that do not exist on the one side or the other. In order to actually list files that are different, you could extend the approach by computing a checksum of each file before diffing the lists: diff <(cd dir1; find . -type f -exec sha1sum {} \; | sort) <(cd dir2; find . -type f -exec sha1sum {} \; | sort) . You might want to extract the command to a script as it's getting quite long.
    – danba
    Commented Aug 22, 2022 at 11:05
  • 1
    This is exactly what I was looking for! I was looking for a way to compare file structures of 2 projects. Thanks!
    – tolik518
    Commented Dec 8, 2023 at 8:37
-5
rsync -rvc --delete --size-only --dry-run source dir target dir

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.