0

Similar to my earlier question Delete the row if it contains more than specific number of non numeric values, I have a data that has non-numeric fields: I want to delete those columns (fields) that has more than specific number of non-numeric characters (Off and No Data) in this example. I want to count from 3rd field onwards, i.e, forget the first two rows.

Tag, Description,2015/01/01,2015/01/01 00:01:00,2015/01/01 00:02:00
1827XYZR/KB.SAT,Data from Process Value, 2.10000, No Data, 2.7
1871XYZR/KB.RAT,Data from process value, Off    , 2.87583, No Data
1962XYMK/KB.GAT,Data from Process Value, No Data, 5      , 3
1867XYST/KB.FAT,Data from process value, 1.05000, 5.87   , 7.80
1871XKZR/KB.VAT,Data from process value, No Data, Off    , 2

In the data above (there are no spaces, I have kept it only to show the formatting), I want to delete all those columns that has more than two (3 or more) occurrence of non-numeric characters. Here, Column 3 has two No Data and one Off, so I want to delete the 3rd Column here. I tried this in a loop without success

awk -F ',' '{
for(n=3; n<=NF; n++)
    {   
        a = $n
        if(n!=NF)
        {
            fmt="%s,";
        }               
        else
        {
            fmt="%s\n";
        }
        if(gsub(/No Data|Off/,"",a)<2)
        {   
            printf(fmt,$n);
        }       
    }
 }' testfile.txt

I also tried

awk -F, '{for(i=3;i<=NF;i++)if(gsub(/No Data|Off/,"",$i)<3)f=f?f FS $i:$i;print f;f=""}' testfile.txt

but this only deletes the the words No Data and Off but keeps the fields (columns) intact. My desired output is:

Tag, Description,2015/01/01 00:01:00,2015/01/01 00:02:00
1827XYZR/KB.SAT,Data from Process Value, Off     ,2.7
1871XYZR/KB.RAT,Data from process value, 2.87583 ,No Data
1962XYMK/KB.GAT,Data from Process Value, 5       ,3
1867XYST/KB.FAT,Data from process value, 5.87    ,7.80
1871XKZR/KB.VAT,Data from process value, Off     ,2
4
  • You have been asking this before
    – hek2mgl
    Sep 19, 2016 at 10:00
  • @hek2mgl It's not the same; in the referred duplicate he would like to remove rows, in this one columns. It is completely different problem. Sep 19, 2016 at 11:47
  • 1
    @JamesBrown Missed that. Question re-opened
    – hek2mgl
    Sep 19, 2016 at 12:00
  • Yea, it seem familiar to me too. Sep 19, 2016 at 13:36

1 Answer 1

1

You can use this awk command:

awk -F, '{
   for (i=1; i<=NF; i++) {
      cell[NR,i]=$i
      if (i>2 && $i+0 != $i)
         nn[i]++
   }
   nf=NF
   nr=NR
} 
END {
   for (r=1; r<=nr; r++) {
      for (c=1; c<=nf; c++)
         if (nn[c]<4)
            printf "%s%s", (c>1?FS:""), cell[r,c]
      print ""
   }
}' file

Tag,Description,2015/01/01 00:01:00,2015/01/01 00:02:00
1827XYZR/KB.SAT,Data from Process Value,No Data,2.7
1871XYZR/KB.RAT,Data from process value,2.87583,No Data
1962XYMK/KB.GAT,Data from Process Value,5      ,3
1867XYST/KB.FAT,Data from process value,5.87   ,7.80
1871XKZR/KB.VAT,Data from process value,Off    ,2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.