163

I have a utility script in Python:

#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
  if line in unique_lines:
    duplicate_lines.append(line)
  else:
    unique_lines.append(line)
    sys.stdout.write(line)
# optionally do something with duplicate_lines

This simple functionality (uniq without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?

Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.

3
  • 4
    Unrelated: you should really use a set rather than a list in that Python script; checking for membership in a list is a linear-time operation. Jul 17, 2012 at 23:18
  • I removed "Python" from your tags and title since this really has nothing to do with Python. Jul 17, 2012 at 23:20
  • 1
    if this had to be done in Python a better approach would involve using the uniq_everseen itertools recipe: docs.python.org/library/itertools.html#recipes
    – iruvar
    Jul 23, 2012 at 17:02

8 Answers 8

391

The UNIX Bash Scripting blog suggests:

awk '!x[$0]++'

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.

20
  • 12
    For a short awk statement like this (no curly brackets involved), the command is simply telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, we are incrementing a node of the array named x and printing the line if the content of that node was not (!) previously set. Dec 17, 2012 at 14:59
  • 11
    Most compact and finest scripts I tumbled across. Kudos! Dec 21, 2015 at 5:46
  • 54
    Surely it would be less obfuscated to name that array e.g. seen instead of x, to avoid giving newbies the impression that awk syntax is line noise Dec 21, 2015 at 10:43
  • 7
    Keep in mind that this will load the entire file into memory, so don't try this on a 3GB text file without lots of RAM to spare. Jun 2, 2017 at 15:39
  • 10
    @Hitechcomputergeek This won't necessarily load the whole file into memory, only the unique lines. This of course could end up being the whole file though if all the lines are unique.
    – deltaray
    Jul 11, 2018 at 17:33
105

A late answer - I just ran into a duplicate of this - but perhaps worth adding...

The principle behind @1_CR's answer can be written more concisely, using cat -n instead of awk to add line numbers:

cat -n file_name | sort -uk2 | sort -n | cut -f2-
  • Use cat -n to prepend line numbers
  • Use sort -u remove duplicate data (-k2 says 'start at field 2 for sort key')
  • Use sort -n to sort by prepended number
  • Use cut to remove the line numbering (-f2- says 'select field 2 till end')
8
  • 4
    Easy to understand, and this is often valuable. Any ideas of performance with big files against shortest Michael Hoffman's solution above? Jan 1, 2015 at 2:50
  • 4
    More readable/maintainable. Needed the same but with a reverse sort to keep only the last occurrence of each unique value. Using both --reverse and --unique in the same sort command doesn't return the results one might expect. Apparently, sort does a premature optimization by 1st applying --unique on the input (in order to reduce processing in subsequent steps). This removes data needed for the --reverse step too early. To fix this, insert a sort --reverse -k2 as the 1st sort in the pipeline: cat -n file_name | sort -rk2 | sort -uk2 | sort -nk1 | cut -f2- Apr 24, 2017 at 9:36
  • 2
    Took just 60 seconds for a 900MB+ text file with so many (randomly placed) duplicate lines that the result is only 39KB. Sufficiently fast.
    – ynn
    Jul 24, 2019 at 14:09
  • "Pipe" version: cat file_name | cat -n | sort -uk2 | sort -nk1 | cut -f2-. Jan 15, 2020 at 18:01
  • 3
    "Pipe" version for keeping last occurrence instead of first one: cat file_name | cat -n | sort -rk2 | sort -uk2 | sort -nk1 | cut -f2-. Jan 15, 2020 at 18:02
10

To remove duplicate from 2 files :

awk '!a[$0]++' file1.csv file2.csv
6

uq

uq is a small tool written in Rust. It performs uniqueness filtering without having to sort the input first, therefore can apply on continuous stream.

There are two advantages of this tool over the top-voted awk solution and other shell-based solutions:

  1. uq remembers the occurence of lines using their hash values, so it doesn't use as much memory use when the lines are long.
  2. uq can keep the memory usage constant by setting a limit on the number of entries to store (when the limit is reached, there is a flag to control either to override or to die), while the awk solution could run into OOM when there are too many lines.
1
  • 1
    Quite inconvenient and less portable, given awk already does this. Feb 9, 2020 at 22:27
5

Michael Hoffman's solution above is short and sweet. For larger files, a Schwartzian transform approach involving the addition of an index field using awk followed by multiple rounds of sort and uniq involves less memory overhead. The following snippet works in bash

awk '{print(NR"\t"$0)}' file_name | sort -t$'\t' -k2,2 | uniq --skip-fields 1 | sort -k1,1 -t$'\t' | cut -f2 -d$'\t'
1
  • this seems to be rather slow, though
    – galois
    Aug 24, 2015 at 6:43
2

Thanks 1_CR! I needed a "uniq -u" (remove duplicates entirely) rather than uniq (leave 1 copy of duplicates). The awk and perl solutions can't really be modified to do this, your's can! I may have also needed the lower memory use since I will be uniq'ing like 100,000,000 lines 8-). Just in case anyone else needs it, I just put a "-u" in the uniq portion of the command:

awk '{print(NR"\t"$0)}' file_name | sort -t$'\t' -k2,2 | uniq -u --skip-fields 1 | sort -k1,1 -t$'\t' | cut -f2 -d$'\t'
-1

I just wanted to remove all duplicates on following lines, not everywhere in the file. So I used:

awk '{
  if ($0 != PREVLINE) print $0;
  PREVLINE=$0;
}'
1
  • 13
    doesn't uniq do just that... Nov 9, 2016 at 11:22
-3

the uniq command works in an alias even http://man7.org/linux/man-pages/man1/uniq.1.html

Not the answer you're looking for? Browse other questions tagged or ask your own question.