-3

inputfile.log

HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Session not available
HSDA_1_BuyVoice): Session not available 
HSDA_1_BuyVoice): Unable to locate element:
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0

output expected

HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_1_BuyVoice): Session not available 
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0
1
  • 1
    it looks like you are expecting us to write a script for you Jan 13, 2012 at 8:54

6 Answers 6

3

Using awk, I would:

  1. Set field separator to ':'
  2. For each row where the saved value of the first field is not equal to the first field:
    • print that entire row
    • save the value of the first field

In other words:

awk -F':' ' $1 != first { print; first = $1 } inputfile.log

I'd be done in time to curl up with a good book.

3
  • Guess I need that book more than anything else. :( Jan 13, 2012 at 9:22
  • With which part of this are you having difficulty?
    – johnsyweb
    Jan 13, 2012 at 9:52
  • 1
    No, it's just that its an excellent solution. +1 cheers!! Jan 13, 2012 at 9:56
2

I did a "translation" for @Johnsyweb's answer:

awk -F':' '($1 in a){next;} !a[$1]++;'  input

oh, forgot the test:

kent$  echo "HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Session not available
HSDA_1_BuyVoice): Session not available 
HSDA_1_BuyVoice): Unable to locate element:
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0
"|awk -F':' '($1 in a){next;} !a[$1]++;'

HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_1_BuyVoice): Session not available 
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0
6
  • Ok you don't have to rub it on my face!! :( Sniff Sniff Jan 13, 2012 at 9:24
  • @JaypalSingh I didn't put any thinking, just did a word2Code translation...
    – Kent
    Jan 13, 2012 at 9:28
  • Yea yea … Though, I never stop learning from your answers. +1 Jan 13, 2012 at 9:29
  • But you have to give me credit. I can make simple things complicated. Isn't it?? :) Jan 13, 2012 at 9:30
  • Not an exact translation, I would only store one 'key' at a time.
    – johnsyweb
    Jan 13, 2012 at 9:55
1

This should work -

awk -F_ '!a[$2]++{b[NR]=$0}
END{for(NR=1;NR<=length(b);NR++) if(b[NR]!~/^$/) print b[NR]}' filename

Test:

[jaypal:~/Temp] cat file
HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Session not available
HSDA_1_BuyVoice): Session not available 
HSDA_1_BuyVoice): Unable to locate element:
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0

[jaypal:~/Temp] awk -F_ '!a[$2]++{b[NR]=$0}END{for (NR=1;NR<=length(b);NR++) if (b[NR]!~/^$/) print b[NR]}' file
HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_1_BuyVoice): Session not available 
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0
2
  • well short codes don't always mean efficient. but very often not easy to understand. yours is not complicated either. it would be better if don't save whole line in array though, but it makes no difference if the input data is so small as the example.
    – Kent
    Jan 13, 2012 at 9:42
  • I agree. Coz the numbers I am indexing on can repeat for different contexts. Jan 13, 2012 at 9:45
1

This might work for you:

 sed ':a;$!N;/^\([^:]*:\).*\n\1/s/\n.*//;ta;P;D' file
0

Probably not very optimized script, it takes two arguments (log file and output file).

#!/bin/sh

file=$1
output=$2

while read line; do
    word=$(echo $line | awk '{ print $1 }')
    grep "$word" $output > /dev/null 2>&1
    if [ $? -ne 0 ]; then
        echo $line >> $output
    fi
done < $file
0

One way using sed:

Content of script.sed:

## Set label 'a'
:a

## While not end of file...
$! {

    ## If pattern space only has one line, read next one.
    /\n/! N

    ## If both lines have same characters until first colon, delete
    ## second one.
    /^\([^:]*\):.*\n\1/ s/\n.*$//

    ## If last substitution command (delete repeated line) go to
    ## label 'a' without reading next input line.
    ta  

    ## We are here because last substitution command failed. It
    ## means that first line is different from second one so 
    ## 'print && delete' first one and save second one to compare in 
    ## next loop.
    P   
    D   
}

## This point should be only at end of file, and default action is 'print' so
## it will print last line left in pattern space.

Execution of the script:

sed -f script.sed infile

And result:

HSDA_17_VerifyShippingOrderNoInOrderConfirmationTest): Timed out after 90 seconds
HSDA_1_BuyVoice): Session not available 
HSDA_9_CreateAccounAndBuyDeviceInCheckoutTest): Timed out after 90 seconds
HSDA_2_BuyMapswithNewUserAtCheckOutTest): Address already in use: connect
HSDA_69_ChangeBillingAddressInNonUSLocaleForAllFieldsTest): Index: 0, Size: 0

Not the answer you're looking for? Browse other questions tagged or ask your own question.