17

I'm stuck with this for several hours now and cycled through a wealth of different tools to get the job done. Without success. It would be fantastic, if someone could help me out with this.

Here is the problem:

I have a very large CSV file (400mb+) that is not formatted correctly. Right now it looks something like this:

This is a long abstract describing something. What follows is the tile for this sentence."   
,Title1  
This is another sentence that is running on one line. On the next line you can find the title.   
,Title2

As you can probably see the titles ",Title1" and ",Title2" should actually be on the same line as the foregoing sentence. Then it would look something like this:

This is a long abstract describing something. What follows is the tile for this sentence.",Title1  
This is another sentence that is running on one line. On the next line you can find the title.,Title2

Please note that the end of the sentence can contain quotes or not. In the end they should be replaced too.

Here is what I came up with so far:

sed -n '1h;1!H;${;g;s/\."?.*,//g;p;}' out.csv > out1.csv

This should actually get the job done of matching the expression over multiple lines. Unfortunately it doesn't :)

The expression is looking for the dot at the end of the sentence and the optional quotes plus a newline character that I'm trying to match with .*.

Help much appreciated. And it doesn't really matter what tool gets the job done (awk, perl, sed, tr, etc.).

4
  • 1
    Do you have an example that's not domain-specific? I'm having trouble seeing what exactly you want changed.
    – robert
    Dec 22, 2010 at 15:54
  • Robert, I changed the example. I hope that this one is better :)
    – herrherr
    Dec 22, 2010 at 15:59
  • Is your csv really just 2 fields or is that just a simplified example? Dec 22, 2010 at 16:25
  • Right now it just has these 2 fields.
    – herrherr
    Dec 22, 2010 at 17:05

2 Answers 2

20

Multiline in sed isn't necessarily tricky per se, it's just that it uses commands most people aren't familiar with and have certain side effects, like delimiting the current line from the next line with a '\n' when you use 'N' to append the next line to the pattern space.

Anyway, it's much easier if you match on a line that starts with a comma to decide whether or not to remove the newline, so that's what I did here:

sed 'N;/\n,/s/"\? *\n//;P;D' title_csv

Input

$ cat title_csv
don't touch this line
don't touch this line either
This is a long abstract describing something. What follows is the tile for this sentence."
,Title1
seriously, don't touch this line
This is another sentence that is running on one line. On the next line you can find the title.
,Title2
also, don't touch this line

Output

$ sed 'N;/\n,/s/"\? *\n//;P;D' title_csv
don't touch this line
don't touch this line either
This is a long abstract describing something. What follows is the tile for this sentence.,Title1
seriously, don't touch this line
This is another sentence that is running on one line. On the next line you can find the title.,Title2
also, don't touch this line
7
  • Thanks. Your example is working for me, but for some reason it doesn't work with my file. Although the pattern is basically the same. I've uploaded a small example to my Dropbox. Perhaps you have an idea what's wrong here. dl.dropbox.com/u/84641/temp/text.csv
    – herrherr
    Dec 22, 2010 at 17:00
  • 1
    @herrherr ok, found and fixed. In my original code i used \+ to match a whitespace one or more times, what I really wanted is * to match a white space zero or more times.
    – SiegeX
    Dec 22, 2010 at 17:13
  • 1
    @herrherr: I found that this (sed 'N;/\n,/s/\n,/,/' text.csv) worked for me once I'd added a trailing newline to the file downloaded from DropBox. Dec 22, 2010 at 17:16
  • This fails if there's a "don't touch" line after the ",Title1" line. It also doesn't eliminate the spurious quote. Dec 22, 2010 at 20:09
  • @Dennis Good call on both accounts. I completely glazed over the part about removing a trailing quote. As for the odd pairings, a bit of P;D love was all that was needed to shore that up.
    – SiegeX
    Dec 22, 2010 at 20:47
14

Yours works with a couple of small changes:

sed -n '1h;1!H;${;g;s/\."\?\n,//g;p;}' inputfile

The ? needs to be escaped and . doesn't match newlines.

Here's another way to do it which doesn't require using the hold space:

sed -n '${p;q};N;/\n,/{s/"\?\n//p;b};P;D' inputfile

Here is a commented version:

sed -n '
$          # for the last input line
{
  p;             # print
  q              # and quit
};
N;         # otherwise, append the next line
/\n,/      # if it starts with a comma
{
  s/"\?\n//p;    # delete an optional comma and the newline and print the result
  b              # branch to the end to read the next line
};
P;         # it doesn't start with a comma so print it
D          # delete the first line of the pair (it's just been printed) and loop to the top
' inputfile
1
  • Just noticed your post after I responded to your comment. No need for loops and special attention to the last line. Appending P;D to the end of my script is all it needed.
    – SiegeX
    Dec 22, 2010 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.