I'm looking to run a Linux command that will recursively compare two directories and output only the file names of what is different. This includes anything that is present in one directory and not the other or vice versa, and text differences.
7 Answers
From the diff man page:
-q
Report only whether the files differ, not the details of the differences.
-r
When comparing directories, recursively compare any subdirectories found.
Example command:
diff -qr dir1 dir2
Example output (depends on locale):
$ ls dir1 dir2
dir1:
same-file different only-1
dir2:
same-file different only-2
$ diff -qr dir1 dir2
Files dir1/different and dir2/different differ
Only in dir1: only-1
Only in dir2: only-2
-
1Thanks - the
diff
man page in CentOS 7 describes-q
as "report only when files differ", which is less clear than what you wrote.– ChapCommented Dec 12, 2018 at 17:58 -
5This compares the actual content of the files which is often what one wants, however, the rsync answer gives the option to only look at the file names and sizes, and not the content; that is sometimes desirable.– stevebCommented Feb 13, 2019 at 21:18
-
-
Also, can include
-x PATTERN
in command to exclude certain subdirectories. For example,diff -qr repo1 repo2 -x ".git"
will compare two directories but will exclude file paths with ".git" in them.– ViFICommented Apr 16, 2020 at 19:12
You can also use rsync
rsync -rv --size-only --dry-run /my/source/ /my/dest/ > diff.out
-
9
--size-only
will miss files of identical size but different content, e.g. old/version.txt "29a" new/version.txt "29b". Use instead:rsync -ric --dry-run old/ new/
where the "-i" argument allows to obtain the file list directly viarsync -ric --dry-run old/ new/ | cut -d" " -f 2
– iolsmitCommented Mar 24, 2015 at 17:15 -
14This is great if you are only looking for missing files (especially across network shares), because it doesn't compare the contents. This helped me find a few files that failed when migrating to a new NAS. Commented Dec 26, 2015 at 18:26
-
7Be sure to include the trailing slash for paths specified on rsync's command line. WIthout them, this won't work correctly, and rsync will likely just enumerate all file names! Commented Nov 18, 2017 at 0:37
-
2With regard to the comments on not checking the content. This is sometimes desirable, at least as first pass.– stevebCommented Feb 13, 2019 at 21:14
-
@OverZealous Yes, but the odds are good that the size won't be the same to the byte. Still, I would just exclude the
--size-only
to check full contents. Commented Sep 9, 2020 at 21:43
If you want to get a list of files that are only in one directory and not their sub directories and only their file names:
diff -q /dir1 /dir2 | grep /dir1 | grep -E "^Only in*" | sed -n 's/[^:]*: //p'
If you want to recursively list all the files and directories that are different with their full paths:
diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}'
This way you can apply different commands to all the files.
For example I could remove all the files and directories that are in dir1 but not dir2:
diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}' xargs -I {} rm -r {}
The approach of running diff -qr old/ new/
has one major drawback: it may miss files in newly created directories. E.g. in the example below the file data/pages/playground/playground.txt
is not in the output of diff -qr old/ new/
whereas the directory data/pages/playground/
is (search for playground.txt in your browser to quickly compare). I also posted the following solution on Unix & Linux Stack Exchange, but I'll copy it here as well:
To create a list of new or modified files programmatically the best solution I could come up with is using rsync, sort, and uniq:
(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq
Let me explain with this example: we want to compare two dokuwiki releases to see which files were changed and which ones were newly created.
We fetch the tars with wget and extract them into the directories old/
and new/
:
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29d.tgz
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29.tgz
mkdir old && tar xzf dokuwiki-2014-09-29.tgz -C old --strip-components=1
mkdir new && tar xzf dokuwiki-2014-09-29d.tgz -C new --strip-components=1
Running rsync one way might miss newly created files as the comparison of rsync and diff shows here:
rsync -rcn --out-format="%n" old/ new/
yields the following output:
VERSION
doku.php
conf/mime.conf
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php
Running rsync only in one direction misses the newly created files and the other way round would miss deleted files, compare the output of diff:
diff -qr old/ new/
yields the following output:
Files old/VERSION and new/VERSION differ
Files old/conf/mime.conf and new/conf/mime.conf differ
Only in new/data/pages: playground
Files old/doku.php and new/doku.php differ
Files old/inc/auth.php and new/inc/auth.php differ
Files old/inc/lang/no/lang.php and new/inc/lang/no/lang.php differ
Files old/lib/plugins/acl/remote.php and new/lib/plugins/acl/remote.php differ
Files old/lib/plugins/authplain/auth.php and new/lib/plugins/authplain/auth.php differ
Files old/lib/plugins/usermanager/admin.php and new/lib/plugins/usermanager/admin.php differ
Running rsync both ways and sorting the output to remove duplicates reveals that the directory data/pages/playground/
and the file data/pages/playground/playground.txt
were missed initially:
(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq
yields the following output:
VERSION
conf/mime.conf
data/pages/playground/
data/pages/playground/playground.txt
doku.php
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php
rsync
is run with theses arguments:
-r
to "recurse into directories",-c
to also compare files of identical size and only "skip based on checksum, not mod-time & size",-n
to "perform a trial run with no changes made", and--out-format="%n"
to "output updates using the specified FORMAT", which is "%n" here for the file name only
The output (list of files) of rsync
in both directions is combined and sorted using sort
, and this sorted list is then condensed by removing all duplicates with uniq
-
Couldn't you just run it backwards (
diff new/ old/
) to see which directories were deleted?– JacquesCommented Apr 5, 2017 at 8:24 -
Running
diff -qr new/ old/
on the example above with the dokuwiki tars produces the same output asdiff -qr old/ new/
– i.e. you see that the directory is new/missing but not the files in it– iolsmitCommented Apr 5, 2017 at 15:08
On my linux system to get just the filenames
diff -q /dir1 /dir2|cut -f2 -d' '
-
7
-
6I did not mean to impute this to you... ;-p Just as a hint for somebody who does... Commented Dec 5, 2014 at 16:34
-
does not work for me. My directory structure is like below
audit-0.0.234/audit-data-warehouse-0.0.234/ audit-0.0.235/audit-data-warehouse-0.0.235/
Commented Aug 4, 2017 at 9:57 -
1
I have a directory.
$ tree dir1
dir1
├── a
│ └── 1.txt
├── b
│ └── 2.txt
└── c
├── 3.txt
├── 4.txt
└── d
└── 5.txt
4 directories, 5 files
I have another directory.
$ tree dir2
dir2
├── a
│ └── 1.txt
├── b
└── c
├── 3.txt
├── 5.txt
└── d
└── 5.txt
4 directories, 4 files
I can diff two directories.
$ diff <(cd dir1; find . -type f | sort) <(cd dir2; find . -type f| sort)
--- /dev/fd/11 2022-01-21 20:27:15.000000000 +0900
+++ /dev/fd/12 2022-01-21 20:27:15.000000000 +0900
@@ -1,5 +1,4 @@
./a/1.txt
-./b/2.txt
./c/3.txt
-./c/4.txt
+./c/5.txt
./c/d/5.txt
-
2This approach is nice, but only lists files that do not exist on the one side or the other. In order to actually list files that are different, you could extend the approach by computing a checksum of each file before diffing the lists:
diff <(cd dir1; find . -type f -exec sha1sum {} \; | sort) <(cd dir2; find . -type f -exec sha1sum {} \; | sort)
. You might want to extract the command to a script as it's getting quite long.– danbaCommented Aug 22, 2022 at 11:05 -
1This is exactly what I was looking for! I was looking for a way to compare file structures of 2 projects. Thanks!– tolik518Commented Dec 8, 2023 at 8:37