149

I have two files (let's say a.txt and b.txt), both of which has a list of names. I have already run sort on both the files.

Now I want to find lines from a.txt which are not present in b.txt.

(I spent lot of time to find the answer for this question, so documenting it for future reference)

0

4 Answers 4

244

The command you have to use is not diff but comm

/usr/bin/comm -23 a.txt b.txt

By default, comm outputs 3 columns: left-only, right-only, both. The -1, -2 and -3 switches suppress these columns.

So, -23 hides the right-only and both columns, showing the lines that appear only in the first (left) file.

If you want to find lines that appear in both, you can use -12, which hides the left-only and right-only columns, leaving you with just the both column.

2
  • 36
    I will add that this works only if both files are sorted. (I know the OP mentioned he sorted the files, but many people, me included, read the question title and then jump to the answers)
    – user247866
    Commented Apr 4, 2014 at 18:06
  • 12
    @user247866: Fortunately comm is kind enough to tell you if they are not sorted :)
    – marlar
    Commented Feb 29, 2016 at 10:48
46

The simple answer did not work for me because I didn't realize comm matches line for line, so duplicate lines in one file will be printed as not-existing in the other. For example, if file1 contained:

Alex
Bill
Fred

And file2 contained:

Alex
Bill
Bill
Bill
Fred

Then comm -13 file1 file2 would output:

Bill
Bill

In my case, I wanted to know only that every string in file2 existed in file1, regardless of how many times that line occurred in each file.

Solution 1: use the -u (unique) flag to sort:

comm -13 <(sort -u file1) <(sort -u file2)

Solution 2: (the first "working" answer I found) from unix.stackexchange:

fgrep -v -f file1 file2

Note that if file2 contains duplicate lines that don't exist at all in file1, fgrep will output each of the duplicate lines. Also note that my totally non-scientific tests on a single laptop for a single (fairly large) dataset showed Solution 1 (using comm) to be almost 5 times faster than Solution 2 (using fgrep).

2
  • 1
    I had my files sorted and passed through uniq. Anyways thanks for the other solutions.
    – Sudar
    Commented Oct 1, 2014 at 7:04
  • The fgrep version will be very slow, if you have tens of thousands of lines.
    – Kai Petzke
    Commented Dec 15, 2021 at 6:58
20

I am not sure why it has been said diff should not be used. I would use it to compare the two files and then output only lines that are in the left file but not in right one. Such lines are flagged by diff with < so it suffices to grep that symbol at the beginning of the line

diff a.txt b.txt  | grep \^\<
2
  • 4
    You can use diff --new-line-format= --unchanged-line-format= a.txt b.txt to suppress the printing of new and unchanged lines. Commented Apr 4, 2017 at 5:36
  • diff worked fine for me. I am on win10, no comm installed. Commented Oct 27, 2021 at 6:13
16

In the case the files wouldn't be sorted yet, you can use:

comm -23 <(sort a.txt) <(sort b.txt)
1
  • 3
    This allocated like 15GB of memory for me for a couple files each < 300 MB...
    – user541686
    Commented Jan 13, 2019 at 17:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.