37

The sort command lets me put lines in alphabetical order and remove duplicate lines. I need something similar that can sort the words on a single line, put them in order, and remove any duplicates. Is there a command for this?

E.g.:

zebra ant spider spider ant zebra ant

Changes to:

ant spider zebra

There is no space before the first word or after the last word.

7 Answers 7

80

This works for me:

$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs
ant spider zebra

You can transform a list of words in a single row to a single column with xargs -n1 , use sort -u and transform back to a single row with xargs.

0
32

The shell was built to parse [:blank:] seperated word lists already. Therefore the use of xargs is completely redundant. The "unique" stuff can be done but its just easier to use sort.

echo $(printf '%s\n' zebra ant spider spider ant zebra ant | sort -u)

3
  • 1
    Accepted answer uses 3 pipes but this one only uses 1. +1
    – akhan
    Apr 17, 2018 at 18:52
  • 2
    I like this answer. Can you expand on the need for 'echo' ? Can you not just do: printf '%s\n' zebra ant spider spider ant zebra ant | sort -u
    – mike
    Jan 13, 2019 at 19:22
  • 1
    Ah. I figured it out. It's to get them back into a string again instead of being multi-line. I'll leave my comment for the next person that doesn't immediately figure it out.
    – mike
    Jan 13, 2019 at 19:25
10

Use tr to change spaces to new lines, then sort, and finally change new lines back to spaces.

echo $(tr ' ' '\n' <<< "zebra ant spider spider ant zebra ant" | sort -u)
1
  • Better than using printf. Could drop the extra tr at the end though. +1
    – akhan
    Apr 17, 2018 at 18:56
7

All of the answers prior to this one can only sort a single line at time. The following can be used to pipe a whole list of such lines into and it will print the sorted list of unique words for each line.

awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }'

Thanks @jaypai for a lot of the syntax used in this.

Example:

>cat file
group label wearable edit_group edit_group_order label_max camera_elevation camera_distance name label_min label_max value_min value_max camera_angle camera_elevation id
id group label wearable edit_group clothing_morph value_min value_max name value_default clothing_morph group
id label show_simple wearable name edit_group edit_group_order group clothing_morph clothing_morph camera_distance label_min label_max value_min value_max camera_distance camera_angle
id group label wearable name edit_group clothing_morph value_min value_max value_default
group label wearable id clothing_morph edit_group edit_group_order label_min label_max value_min value_max name camera_distance camera_angle camera_elevation
id group label wearable edit_group name label_min label_max value_min value_max wearable
name id group wearable edit_group id group wearable id group wearable id group wearable value_min value_max

>cat file | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }'
camera_angle camera_distance camera_elevation edit_group edit_group_order group id label label_max label_min name value_max value_min wearable 
clothing_morph edit_group group id label name value_default value_max value_min wearable 
camera_angle camera_distance clothing_morph edit_group edit_group_order group id label label_max label_min name show_simple value_max value_min wearable 
clothing_morph edit_group group id label name value_default value_max value_min wearable 
camera_angle camera_distance camera_elevation clothing_morph edit_group edit_group_order group id label label_max label_min name value_max value_min wearable 
edit_group group id label label_max label_min name value_max value_min wearable 
edit_group group id name value_max value_min wearable
2
  • Good, but adds a dangling space to each line; can be fixed by changing the last loop to for (i=1; i<n; i++) printf b[i]" "; print b[n] Apr 21, 2018 at 12:54
  • Note, the function asorti seems to be unique to the GNU version of awk. To make this work on MacOS, install GNU awk with brew install gawk, and use gawk instead of awk. May 12 at 12:15
2

Use python

$ echo "zebra ant spider spider ant zebra ant" | python -c 'import sys; print(" ".join(sorted(set(sys.stdin.read().split()))))'
ant spider zebra
2

Using perl:

perl -lane '
  %a = map { $_ => 1 } @F;
  print join qq[ ], sort keys %a;
' <<< "zebra ant spider spider ant zebra ant"

Result:

ant spider zebra
1

Using awk:

awk '{for(i=1;i<=NF;i++) a[$i]++} END{for(i in a) printf i" ";print ""}' INPUT_FILE

Test:

[jaypal:~/Temp] cat file
zebra ant spider spider ant zebra ant
[jaypal:~/Temp] awk '{for (i=1;i<=NF;i++) a[$i]++} END{for (i in a) printf i" ";print ""}' file
zebra spider ant 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.