0

I'm trying to convert a file's encoding and replace some text along the way. Unfortunately, I'm getting an OutOfMemory exception. I'm not sure why. As I understand it, it streams the original file line by line into a var (str), completes a couple of string replacements, and then writes the converted line to the StreamWriter.

Can someone tell me what I'm doing wrong here?

EDIT 1 - I'm currently testing a single file - 1GB:2.5m rows. - Replaced read and replace into a single line. Same results!

EDIT 2

???By the way, can anyone tell me why the question was downgraded? I'd like to know for future postings.???

The problem is with the file itself. It's output from SQL Server BCP where I explicitly flag the row terminator with a specific string. By default, when the row terminator flag is omitted, BCP adds a newline at the end of each row and the code below works perfectly.

What I still don't understand is: when I set the row terminator flag with a specific string, each record appears on a newline, so why doesn't streamreader see each record on a separate line? Instead, it appears it views the entire file as one long line. That still doesn't explain the OOM exception since I have well over a 100G of memory.

Unfortunately, explicitly setting the row terminator flag is a must. For now, I'll take this over to dba exchange.

Thanks

static void Main(string[] args)
    {           
        String msg = String.Empty;
        String str = String.Empty;
        DirectoryInfo dInfo = new DirectoryInfo(@"\\server\share");
        foreach (var f in dInfo.GetFiles())
        {
            using (StreamReader sr = new StreamReader(f.FullName, Encoding.Unicode, false))
            {
                using (StreamWriter sw = new StreamWriter(f.DirectoryName + "\\new\\" + f.Name, false, Encoding.UTF8))
                {
                    try
                    {    
                        while (!sr.EndOfStream)
                        {
                            str = sr.ReadLine().Replace("this","that");
                            sw.WriteLine(str);
                        }
                    }
                    catch (Exception e)
                    {
                        msg += f.Name + ": " + e.Message;
                    }
                }
            }
        }
        Console.WriteLine(msg);
        Console.ReadLine();                
    }
4
  • How big are the files in the directory?
    – Black Frog
    Apr 27, 2014 at 21:11
  • 1
    Well, from this code this is not your answer, but the output of str.Replace needs to be assigned to another string. Replace doesn't work in place.
    – Steve
    Apr 27, 2014 at 21:11
  • @Steve - Thanks. I updated it to a single line but the results are still the same. Apr 27, 2014 at 21:37
  • Just to isolate the problem. If you remove the replace do you still have the problem?
    – Steve
    Apr 27, 2014 at 22:03

1 Answer 1

0

Well, you're main reading and writing code needs just one line of data. Your msg string, on the other hand, keeps getting larger and larger with each exception.

You'll need to have many millions of files in the folder to get an OutOfMemory exception this way, though.

4
  • Indeed! However, if my code were correct, the exception would be few and far between. Apr 27, 2014 at 21:39
  • Do you have lots of exceptions?
    – zmbq
    Apr 27, 2014 at 21:40
  • No! Just the one that kills it. Apr 27, 2014 at 21:54
  • What's the memory usage of the process? You might have to go John Robbins on it and use windbg.
    – zmbq
    Apr 27, 2014 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.