14

Consider:

public static void ConvertFileToUnicode1252(string filePath, Encoding srcEncoding)
{
    try
    {
        StreamReader fileStream = new StreamReader(filePath);
        Encoding targetEncoding = Encoding.GetEncoding(1252);

        string fileContent = fileStream.ReadToEnd();
        fileStream.Close();

        // Saving file as ANSI 1252
        Byte[] srcBytes = srcEncoding.GetBytes(fileContent);
        Byte[] ansiBytes = Encoding.Convert(srcEncoding, targetEncoding, srcBytes);
        string ansiContent = targetEncoding.GetString(ansiBytes);

        // Now writes contents to file again
        StreamWriter ansiWriter = new StreamWriter(filePath, false);
        ansiWriter.Write(ansiContent);
        ansiWriter.Close();
        //TODO -- log success  details
    }
    catch (Exception e)
    {
        throw e;
        // TODO -- log failure details
    }
}

The above piece of code returns an out-of-memory exception for large files and only works for small-sized files.

9
  • 12
    Can you not do it line by line?
    – BugFinder
    Mar 2, 2017 at 9:16
  • 8
    You don't need to read whole contents with ReadToEnd. Read chunk, convert, write, repeat.
    – Evk
    Mar 2, 2017 at 9:18
  • 3
    Use foreach(string line in File.ReadLines(filePath)) ... process line ... Mar 2, 2017 at 9:18
  • 8
    Side note: don't write throw e; but rather only throw; you'll keep your stack trace in tact this way. And please, Dispose your disposables (the Streams) Mar 2, 2017 at 11:19
  • 1
    When OutOfMemoryException is seen on a machine with plenty of available memory, it's a sign that the .Net Runtime could not allocate a single contiguous block of memory large enough to satisfy the request. As containers such as List<T> grow, the underlying arrays double in size each time. I've seen this happen when running X86 (32 bit) code because the address space is limited to 4GB.
    – sevzas
    Mar 2, 2017 at 12:07

3 Answers 3

12

I think still using a StreamReader and a StreamWriter but reading blocks of characters instead of all at once or line by line is the most elegant solution. It doesn't arbitrarily assume the file consists of lines of manageable length, and it also doesn't break with multi-byte character encodings.

public static void ConvertFileEncoding(string srcFile, Encoding srcEncoding, string destFile, Encoding destEncoding)
{
    using (var reader = new StreamReader(srcFile, srcEncoding))
    using (var writer = new StreamWriter(destFile, false, destEncoding))
    {
        char[] buf = new char[4096];
        while (true)
        {
            int count = reader.Read(buf, 0, buf.Length);
            if (count == 0)
                break;

            writer.Write(buf, 0, count);
        }
    }
}

(I wish StreamReader had a CopyTo method like Stream does, if it had, this would be essentially a one-liner!)

1
  • Thanks @Matti. This question helps me achieving the task. I could convert encoding of file more than 1.5GB without any exception. Mar 3, 2017 at 6:41
1

Don't readToEnd and read it like line by line or X characters at a time. If you read to end, you put your whole file into the buffer at once.

-1

Try this:

using (FileStream fileStream = new FileStream(filePath, FileMode.Open))
{
    int size = 4096;
    Encoding targetEncoding = Encoding.GetEncoding(1252);
    byte[] byteData = new byte[size];

    using (FileStream outputStream = new FileStream(outputFilepath, FileMode.Create))
    {
        int byteCounter = 0;

        do
        {
            byteCounter = fileStream.Read(byteData, 0, size);

            // Convert the 4k buffer
            byteData = Encoding.Convert(srcEncoding, targetEncoding, byteData);

            if (byteCounter > 0)
            {
                outputStream.Write(byteData, 0, byteCounter);
            }
        }
        while (byteCounter > 0);

        inputStream.Close();
    }
}

Might have some syntax errors as I've done it from memory but this is how I work with large files, read in a chunk at a time, do some processing and save the chunk back. It's really the only way of doing it (streaming) without relying on massive IO overhead of reading everything and huge RAM consumption of storing it all, converting it all in memory and then saving it all back.

You can always adjust the buffer size.

If you want your old method to work without throwing the OutOfMemoryException, you need to tell the Garbage Collector to allow very large objects.

In App.config, under <runtime> add this following line (you shouldn't need it with my code but it's worth knowing):

<gcAllowVeryLargeObjects enabled="true" />
10
  • 4
    That just won't work with all input. The input is in UTF8, and there's no guarantee that by reading exactly 4K bytes you won't read in a partial character that's been encoded in more than one byte. If that happens, it won't be read correctly and you'll have invalid data. Mar 2, 2017 at 9:29
  • I can't see anywhere in the question referring to UTF8, isn't Source Encoding passed in as a parameter? Yeah it will need tweaking for UTF8 but, if your file is all in a single line (to save space by not using unnecessary whitespace or new lines eg. XML) then doing line by line won't work and only way I'm aware of is streaming the file. The buffer size can always be adjusted in each iteration based on partial data being read. Mar 2, 2017 at 9:33
  • The StreamReader(string path) constructor that the OP is using opens the input stream as UTF8. See the linked documentation. In the extremely unlikely event that all the text is on one line, then the correct approach is to use the StreamReader.Read() overload that reads a specified number of characters from a file. NEVER read a fixed sized buffer to read from a file where the characters may have variable-length encoding. It's almost always a bug. Mar 2, 2017 at 9:37
  • As an experiment, try your code with a file produced like this: File.WriteAllText(filePath, new string('x', 4095) + "ÿ"); Mar 2, 2017 at 9:47
  • You'd be surprised how many HUGE files are in a single line provided the format allows it (of course tab or comma separated wouldn't work) but most of the XML files I process are saved to a single line to save on storage costs and transfer costs (especially the indentation.) It's also possible to check if a byte is a single UTF8 character or part of a multi byte character. The answer posted here obviously doesn't do that, and the question never asked for it explicitly. Therefore it's NOT the wrong approach and with UTF8 byte checking it will be a godo way to handle HUGE single line XML files. Mar 2, 2017 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.