8

I've been trying to figure out a way to select lines that are unique based on the values of two columns. For example, here is a sample/example of my file:

chr1    10    12
chr1    10    12
chr1    10    11
chr1    9    12
chr2    15    20

And this is what I want my output to look like:

chr1    10    12
chr1    10    11
chr1    9    12
chr2    15    20

I've tried to use this code because I like how it doesn't mess with the order of my file:

awk -F"\t" '!_[$2]++' SNP_positions.txt > SNP_positions_uniq.txt

However, it only captures unique features of one column. Is there a way to edit this to make it work for two columns?

4
  • you mention two columns but there are three in your file. Which ones should be considered? Give a more generic example if possible. Your current approach works for the given file, so there may be something else.
    – fedorqui
    Jun 17, 2015 at 15:06
  • I want to find lines that have unique values in both column two and three. If I used the code above my output would not give me the 3rd line in my output (chr1 122332 130204). I'll edit to make it more simple Jun 17, 2015 at 15:11
  • Whoever told you to use _ as a variable name is an idiot, do not listen to them any more. We all know that single-letter variable names are best avoided in general as they do nothing to improve the clarity of your code, but using something that's not even a letter is just ridiculous. Jun 17, 2015 at 15:25
  • @cosmictypist how to run this query in mysql. i have the same problem Jun 8, 2017 at 6:46

2 Answers 2

16

You can perfectly use an index that uses more than one field for the array elements:

awk -F"\t" '!seen[$2, $3]++' file

In this case we use $2, $3 as index. This way, we will get all different elements of the tuples ($2, $3).

4
  • If I wanted to use this for all three columns would the code look like this: awk -F"\t" '!uniq[$1 FS $3]++' file or awk -F"\t" '!uniq[$1 FS $2 FS $3]++' file' Jun 17, 2015 at 15:24
  • @christylynn002 If you want to use the full line it is faster to say uniq[$0]. If you want three out of more, uniq[$1 FS $2 FS $3]. Note we use FS to prevent 2\tab3 match a line being 2 3.
    – fedorqui
    Jun 17, 2015 at 15:25
  • Ah, perfect. Thank you! I think I finally understand it. Jun 17, 2015 at 15:27
  • 2
    @jaypalsingh note that the , in an array index is not the same as a , in a print statement. The former gets the value of SUBSEP (a control char by default) while the latter gets the value of OFS (a space by default). Jun 17, 2015 at 15:30
3

The awk solution provided is great and fast, but I came here myself looking for the sort solution, which is probably slower in almost all cases:

cat file | sort -u -k1,1 -k2,2 -k3,3 -s

I found the answer here: https://stackoverflow.com/a/12546627/778533

1
  • You can speed this up by writing LC_ALL=C sort -S70% -u -k1,1 -k2,2 -k3,3 -s
    – miku
    Aug 5, 2021 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.