Is there way to delete duplicate lines in a file in Unix?
I can do it with sort -u and uniq commands. but I want to use sed or awk.
Is that possible?
Join Stack Overflow to learn, share knowledge, and build your career.
|
|||||
|
|
From http://sed.sourceforge.net/sed1line.txt: (Please don't ask me how this works ;-) )
|
|||||||||||||||||||||
|
|
Perl one-liner similar to @jonas's awk solution:
This variation removes trailing whitespace before comparing:
This variation edits the file in-place:
This variation edits the file in-place, and makes a backup
|
||||
|
|
|
The one-liner that Andre Miller posted above works except for recent versions of sed when the input file ends with a blank line and no chars. On my Mac my CPU just spins. Infinite loop if last line is blank and has no chars:
Doesn't hang, but you lose the last line
The explanation is at the very end of the sed FAQ:
|
|||
|
|
|
An alternative way using Vim(Vi compatible): Delete duplicate, consecutive lines from a file:
Delete duplicate, nonconsecutive and nonempty lines from a file:
|
||||
|
|
The first solution is also from http://sed.sourceforge.net/sed1line.txt
the core idea is:
Explains:
The second solution is easy to understood (from myself):
the core idea is:
Explains:
|
||||
|
|
Deletes the duplicate lines using awk. |
|||||||||
|
uniqalone is enough. – Michael Krelin - hacker Sep 18 '09 at 12:59awk, but will be quite resource consuming on bigger files. – Michael Krelin - hacker Sep 18 '09 at 13:00