2

I need to process a lot of Mechanical Turk HITS fast, so I'm trying to write an AWK command/script.

I need it to do three things:

  1. Grab and Print Specific Columns

  2. Remove some text from certain fields.

  3. Output quoted fields.

I want to take my input:

"1","2","Input.image_url","Answer.main"
"1","2","http://i.imgur.com/rGJA3YU.jpg","text"

And get out:

"image_url","main"
"http://i.imgur.com/rGJA3YU.jpg","text"

So far I have

awk -F'","|^"|"$' '{sub("^\"","")} {print $3 ", "$4}' test.csv > output.csv

Which prints:

Input.image_url, Answer.main
http://i.imgur.com/rGJA3YU.jpg, text

How would I alter this?

Thanks for the look, I really appreciate it. I'm terribly new at AWK.

EDIT: This snippet works for the sample I provided, but unfortunately something is wrong. I figured I could simplify the input/output to make the job easier, but it seems I've skipped over something. So I will fill in the details...

When I use:

awk 'BEGIN{FS=OFS=","} {gsub(/"[[:alpha:]]+\./,"\""); print $28, $31}' test.csv > output.csv

For:

"HITId","HITTypeId","Title","Description","Keywords","Reward","CreationTime","MaxAssignments","RequesterAnnotation","AssignmentDurationInSeconds","AutoApprovalDelayInSeconds","Expiration","NumberOfSimilarHITs","LifetimeInSeconds","AssignmentId","WorkerId","AssignmentStatus","AcceptTime","SubmitTime","AutoApprovalTime","ApprovalTime","RejectionTime","RequesterFeedback","WorkTimeInSeconds","LifetimeApprovalRate","Last30DaysApprovalRate","Last7DaysApprovalRate","Input.image_url","Input.main_text","Answer.SEND","Answer.SUBJECT","Answer.main","Approve","Reject"
"373L46LKP7703E3YWZRRTZTZNUJJKX","3H9KHFULG43TZRE1KD4ITGVT4OWCEU","Transcribe the text contained in the image","Transcribe the text contained in the image","transcribe, image, text","$0.01","Mon Aug 25 20:47:26 GMT 2014","1","BatchId:1651513;","900","60","Mon Sep 01 20:47:26 GMT 2014","","","33IZTU6J812191JP8EKV0EN8HD7XS2","A1GOJEDZM2CQTN","Submitted","Mon Aug 25 20:48:15 GMT 2014","Mon Aug 25 20:48:26 GMT 2014","Mon Aug 25 13:49:26 PDT 2014","","","","11","100% (3/3)","100% (3/3)","0% (0/0)","http://i.imgur.com/rGJA3YU.jpg","hippy hay","","","text"

It prints:

"image_url","main"
"100% (3/3)",""

But I need:

"image_url","main"
"http://i.imgur.com/rGJA3YU.jpg","text"

The first row works great but for some reason it's returning another column in the second row.

2 Answers 2

2

You don't state in your question how to identify which text to remove from the fields so this may or may not be what you want:

$ awk 'BEGIN{FS=OFS=","} {gsub(/"[[:alpha:]]+\./,"\""); print $3, $4}' file
"image_url","main"
"http://i.imgur.com/rGJA3YU.jpg","text"
1
  • That worked great for the code I provided, unfortunately it doesn't work for my real application (I edited with an explanation) Sorry to have mucked this up, but could you take another crack at this? Thanks again!
    – debaser
    Aug 25, 2014 at 23:18
0

If data always has same number of fields and two lines:

awk -F, 'NR==1{sub(/[^.]*\./,"\"",$28);split($29,a,"[._]");print $28,"\""a[2]"\""} NR==2 {print $30,$NF}' OFS=, file
"image_url","main"
"http://i.imgur.com/rGJA3YU.jpg","text"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.