12

I have many large csv files (1-10 gb each) which I'm importing into databases. For each file, I need to replace the 1st line so I can format the headers to be the column names. My current solution is:

using (var reader = new StreamReader(file))
{
    using (var writer = new StreamWriter(fixed))
    {
        var line = reader.ReadLine();
        var fixedLine = parseHeaders(line);
        writer.WriteLine(fixedLine);

        while ((line = reader.ReadLine()) != null)
            writer.WriteLine(line);
    }
}

What is a quicker way to only replace line 1 without iterating through every other line of these huge files?

9
  • I'd probably just do this from the command line. copy headerfile+csvfile newfile You could make a batch file with all of the files that need to be changed. (Oh, except it looks like you don't know what the headers will be beforehand. If not, this won't help.) Dec 6, 2012 at 16:41
  • If the tool you're using happens to take input through stdin: instead of a file, you can create a stream composed of the headers and the body and feed that into your import tool when you need it. e.g. cat headerfile bigfile | import_tool Dec 6, 2012 at 16:44
  • This is very sensitive to the Lengths of fixedLine and the first line. Can fixedLine ever be larger? Could line2 move to the end of the file? Dec 6, 2012 at 16:46
  • 2
  • 2

2 Answers 2

7

If you can guarantee that fixedLine is the same length (or less) as line, you can update the files in-place instead of copying them.

If not, you can possibly get a little performance improvement by accessing the .BaseStream of your StreamReader and StreamWriter and doing big block copies (using, say, a 32K byte buffer) to do the copying, which will at least eliminate the time spent checking every character to see if it's an end-of-line character as happens now with reader.ReadLine().

3
  • If it were less, what would you do with the "free" space? Pad it with spaces?
    – Jodrell
    Dec 6, 2012 at 17:08
  • 1
    Probably, but it would depend on the format. In a CSV file, I could quote the header elements, and add padding spaces after commas. Dec 6, 2012 at 17:13
  • 1
    +1. @Jodrell, yes, spaces in most cases are ignored for CSV - so using them for padding would be OK (I tried to suggest it in my identical answer :) ). Spaces also safe for most encodings - so can be written directly into stream as bytes if needed. Dec 6, 2012 at 17:14
6

The only thing that can significantly speed it up is if you can really replace first line. If new first line is no longer than old one - replace (with space padding if needed) the first line carefully.

Otherwise - you have to create new file and copy the rest after first line. You may be able to optimize copying a bit by adjusting buffer sizes/explicit copy as binary/per-allocating size, but it will not change the fact that you need to copy whole file.

One more cheat if you planning to drop CSV data into DB anyway: if order does not matter you can read some lines from the beginning, replace them with new header and add the removed lines to the end of the file.

Side note: if this is one-time operation I'd simply copy files and be done with it... Debugging code that inserts data into middle of text file with potentially different encoding may not worth an effort.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.