3

I have a .sh file that takes a log file and extracts data and makes reports. I would like to calculate what percentage of the total lines does an error pop-up (top talkers).

So far I have this:

awk '// {print $4, substr($0, index($0,$9))}' | sort \
                        | uniq -cd | sort -nr | head -n20 > $filename-sr1.tmp

This outputs two columns, the count followed by the line.

How can I take just the count to make the calculations. Eg. count / total_lines = 0.000000...

2 Answers 2

4

Here's using just awk, though the output order will be arbitrary, so you may want to pipe it to a sort -n

$ cat file
foo
foo
bar
foo
quux
quux
$ awk '{a[$0]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}' file
 33.3%  quux
 50.0%  foo

And adapting your current awk:

awk '{a[$4" "substr($0, index($0,$9))]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}'
# or possibly
awk '{s=$4; for(i=9;i<=NF;++i) s=s" "$i; a[s]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}'
2
  • Thanks for the response...although if you can direct your attention to my reply to the answer above you may be able to help me as well...thank you :)
    – MikeyC343
    Aug 16, 2012 at 19:12
  • Pipe to sort -nr :)
    – Asclepius
    Nov 27, 2013 at 20:30
3

First I looked to get some similar output to you

cat text.txt | sort | uniq -cd | sort -nr | head -n20 > output.txt

output.txt now looks like this:

      5 red
      3 orange
      3 blue
      2 green

Hopefully that is similar to the output you have?

To get the percentages, count the lines in the original file, and then loop though each line in the output file and use cut to snip out each word/phrase count and bc to do the sums:

total_lines=$(wc -l < text.txt)
while read -r line; do 
    count=$(echo $line | cut -f1 -d " "); 
    percent=$(echo "scale=4; ($count/$total_lines)*100" | bc); 
    echo "$percent% -- $line"; 
done < output.txt;

Result looks like:

38.4600% -- 5 red
23.0700% -- 3 orange
23.0700% -- 3 blue
15.3800% -- 2 green
2
  • OK that works...Step 2 is porting over into .bat :/ basically I'm creating reports in a linux and windows environment. You wouldn't be able to help me with a bat version now would you? I cheated a little and installed gawk and coreutils on my windows machine from GNU :p
    – MikeyC343
    Aug 16, 2012 at 19:09
  • @MikeyC343 Then you're just one step closer of installing bash on the windows machine as well... then you can have one script that works on both systems. Then again, you already have awk, so you can just do it all with awk and drop the bash/cmd.exe requirement.
    – geirha
    Aug 16, 2012 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.