The simple answer did not work for me because I didn't realize comm
matches line for line, so duplicate lines in one file will be printed as not-existing in the other. For example, if file1 contained:
Alex
Bill
Fred
And file2 contained:
Alex
Bill
Bill
Bill
Fred
Then comm -13 file1 file2
would output:
Bill
Bill
In my case, I wanted to know only that every string in file2 existed in file1, regardless of how many times that line occurred in each file.
Solution 1: use the -u
(unique) flag to sort
:
comm -13 <(sort -u file1) <(sort -u file2)
Solution 2: (the first "working" answer I found) from unix.stackexchange:
fgrep -v -f file1 file2
Note that if file2 contains duplicate lines that don't exist at all in file1, fgrep
will output each of the duplicate lines. Also note that my totally non-scientific tests on a single laptop for a single (fairly large) dataset showed Solution 1 (using comm
) to be almost 5 times faster than Solution 2 (using fgrep
).