If you're running Bash version 4 or above (which should be the case in any modern version of Linux), you can get unique array values in bash by creating a new associative array that contains each of the values of the original array. Something like this:
$ a=(aa ac aa ad "ac ad")
$ declare -A b
$ for i in "${a[@]}"; do b["$i"]=1; done
$ printf '%s\n' "${!b[@]}"
ac ad
ac
aa
ad
This works because in any array (associative or traditional, in any language), each key can only appear once. When the for
loop arrives at the second value of aa
in a[2]
, it overwrites b[aa]
which was set originally for a[0]
.
Doing things in native bash can be faster than using pipes and external tools like sort
and uniq
, though for larger datasets you'll likely see better performance if you use a more powerful language like awk, python, etc.
If you're feeling confident, you can avoid the for
loop by using printf
's ability to recycle its format for multiple arguments, though this seems to require eval
. (Stop reading now if you're fine with that.)
$ eval b=( $(printf ' ["%s"]=1' "${a[@]}") )
$ declare -p b
declare -A b=(["ac ad"]="1" [ac]="1" [aa]="1" [ad]="1" )
The reason this solution requires eval
is that array values are determined before word splitting. That means that the output of the command substitution is considered a single word rather than a set of key=value pairs.
While this uses a subshell, it uses only bash builtins to process the array values. Be sure to evaluate your use of eval
with a critical eye. If you're not 100% confident that chepner or glenn jackman or greycat would find no fault with your code, use the for loop instead.