7

Possible Duplicate:
How to read a large (1 GB) txt file in .NET?

What is the optimal way in C# to read an file, replace some strings and write in another new file? I need to do this with very large files like 8GB or 25GB.

1
  • plain text, db backup for sanitize.
    – iLevi
    Apr 13, 2012 at 21:21

3 Answers 3

11

There isn't much you can optimize about the I/O, most of the optimization should be on the string comparison to determine if the string should be replaced or not, basically you should do this

protected void ReplaceFile(string FilePath, string NewFilePath)
{
   using (StreamReader vReader = new StreamReader(FilePath))
   {
      using (StreamWriter vWriter = new StreamWriter(NewFilePath))
      {
         int vLineNumber = 0;
         while (!vReader.EndOfStream)
         {
            string vLine = vReader.ReadLine();
            vWriter.WriteLine(ReplaceLine(vLine, vLineNumber++));
         }
      }
   }
}
protected string ReplaceLine(string Line, int LineNumber )
{
   //Do your string replacement and 
   //return either the original string or the modified one
   return Line;
}

What is your criteria to find and replace a string?

2
  • If you don't need the line number, you should remove it from the ReplaceLine() declaration and from the calling to that function
    – oamilkar
    Apr 13, 2012 at 23:05
  • Great answer. Shouldn't cause an exception from reading too much into memory per line. Oct 29, 2015 at 22:03
2

I did 200GB files before. Please use StreamReader and StreamWriter.

StreamReader

StreamWriter

0
1

Are there lines that are not overly large? If so, you can read in the file line by line, do the replace on that line, and then write out that line to the new file. Since it's streamed very little memory is required.

Not the answer you're looking for? Browse other questions tagged or ask your own question.