6

I have two files like shown below which are tab-delimited:

file A

chr1   123 aa b c d
chr1   234 a  b c d
chr1   345 aa b c d
chr1   456 a  b c d
....

file B

xxxx  abcd    chr1   123    aa    c    d    e
yyyy  defg    chr1   345    aa    e    f    g
...

I want to join the two files based on 3 columns with "chr1", "123" and "aa" and add first two columns from file B to file A, such that output looks as shown below: output:

chr1   123    aa    b    c    d    xxxx    abcd
chr1   234    a     b    c    d
chr1   345    aa    b    c    d    yyyy    defg
chr1   456    a    b    c    d

Could anyone help to do this in awk. If possible using awk oneliners?

1

2 Answers 2

12
+50

Here is one approach using awk:

$ awk 'NR==FNR{a[$3,$4]=$1OFS$2;next}{$6=a[$1,$2];print}' OFS='\t' fileb filea
chr1    123     a    b    c     xxxx    abcd
chr1    234     a    b    c 
chr1    345     a    b    c     yyyy    defg
chr1    456     a    b    c 

Explanation:

NR==FNR             # current recond num match the file record num i.e in filea
a[$3,$4]=$1OFS$2    # Create entry in array with fields 3 and 4 as the key
next                # Grab the next line (don't process the next block)
$6=a[$1,$2]         # Assign the looked up value to field 6 (+rebuild records)  
print               # Print the current line & the matching entry from fileb ($6)

OFS='\t'            # Seperate each field with a single TAB on output

Edit:

For the 3 field problem you simple add the extra field:

$ awk 'NR==FNR{a[$3,$4,$5]=$1OFS$2;next}{$6=a[$1,$2,$3];print}' OFS='\t' fileb filea
chr1    123    aa     b      c     xxxx     abcd
chr1    234    a      b      c  
chr1    345    aa     b      c     yyyy     defg
chr1    456    a      b      c 
6
  • i have modified the original problem. Coudl you provide solution for that.
    – chas
    Aug 6, 2013 at 20:42
  • Thanks. I was adding the extra field in the following way which went wrong:awk 'NR==FNR{a[$3,$4,$5]=$1OFS$2OFS$3;next}{$6=a[$1,$2];print}' OFS='\t' fileb filea.
    – chas
    Aug 6, 2013 at 20:54
  • 1
    +1 but you should set FS as well in case there's any spaces within the fields. Stunning misunderstanding of the script by the OP though! Aug 7, 2013 at 1:02
  • @sudo_O fileb is very huge file and when im trying to run the code it gives the error: "more_nodes: nextfree: can't allocate 6400 bytes of memory (Cannot allocate memory)". Could you modify the script so that fileb is read in chunks to solve the memory problem ?
    – chas
    Aug 20, 2013 at 12:09
  • @user1779730 this is what the command split does. Read man split. Aug 20, 2013 at 12:13
2

You can use join, but the pipeline gets so complicated it might be easier to switch to a more powerful language like Perl.

join -11 -21 -o1.1,1.2,1.3,1.4,1.5,2.4,2.5 \
     <(sed 's/ \+/:/' fileA | sort) \
     <(sed 's/ \+/:/' fileB | sort) \
 | join -11 -22 -a1 -o1.1,1.2,1.3,1.4,1.5,1.6,1.7,2.5,2.6 \
     - <(sed 's/ \+\([^ ]\+\) \+\([^ ]\+\)/ \1:\2/' fileC | sort -k2) \
 | sed 's/:/ /'

Perl solution, using a hash to remember all the information:

#!/usr/bin/perl
use warnings;
use strict;

#             key_start  key_end  keep_from  output
my %files = (A => [0,      1,      2,       [0 .. 3]],
             B => [0,      1,      2,       [-2, -1]],
             C => [1,      2,      3,       [-2, -1]],
            );

my %hash;

for my $file (keys %files) {
    open my $FH, '<', "file$file" or die "file$file: $!";
    while (<$FH>) {
        my @fields = split;
        $hash{"@fields[$files{$file}[0], $files{$file}[1]]"}{$file}
            = [ @fields[$files{$file}[2] .. $#fields] ];
    }
}

for my $key (sort keys %hash) {
    print $key, join(' ', q(),
                     grep defined, map {
                         @{ $hash{$key}{$_} }[@{ $files{$_}[-1] }]
                     } sort keys %files), "\n";
}
1
  • @user1779730: Perl solution added.
    – choroba
    Nov 11, 2012 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.