4

I'm using CSVHelper in my c# project and reading a big csv data file (about 2000 records) into memory.

https://github.com/JoshClose/CsvHelper

it works fine if the records are under 500, it always throw me IOException on different stage depends on the network or if the number goes up. I currently deployed to the Azure cloud platform, so the reading from blob storage to server, that shouldn't be any network problem.

CsvHelper.CsvParserException: A parsing error occurred.

Row: '995' (1 based)
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
   at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
   at CsvHelper.CsvParser.GetChar(Int32& fieldStartPosition, Int32& rawFieldStartPosition, String& field, Boolean prevCharWasDelimiter, Int32& recordPosition, Int32& fieldLength, Boolean isPeek) in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 445
   at CsvHelper.CsvParser.ReadLine() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 247
   at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 108
   --- End of inner exception stack trace ---
   at CsvHelper.CsvParser.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvParser.cs:line 136
   at CsvHelper.CsvReader.Read() in c:\Projects\CsvHelper\src\CsvHelper\CsvReader.cs:line 173

it throws on while (csv.read())

 var wc = new WebClient();

            using (var sourceStream = wc.OpenRead(fileUrl))
            {

                using (var csv = new CsvReader(new StreamReader(sourceStream)))
                {


                    while (csv.Read())
                    {
                        try
                        {

//some reading operation
}
 catch (Exception ex)
                        {
                            _logger.Error(ex);
                        }
                    }
                    _logger.InfoFormat("Finished {0} reading data #{1}");

                }
            }

Anywhere to set streamreader timeout value?

3
  • 1
    Just a thought but like so many other things in Azure, should you have some retry logic around csv.Read() to handle transient errors? Also, if you really think there are network-triggered issues (other than rare events, I'd doubt that's truly the source of your issues), then you could/should grab the file and cache it on the local drive as quickly as possible, then stream from there.
    – Jaxidian
    Apr 15, 2013 at 16:57
  • you mean copy the file from blob storage to local server and parse it from there?
    – Kiddo
    Apr 15, 2013 at 17:14
  • yes. Copy locally and then parse it. Or just copy to local Memory, no need to save it, but make sure you have the whole content first.
    – astaykov
    Mar 7, 2014 at 9:39

1 Answer 1

2
+100

When working with Cloud resources (be it Azure or any other Cloud Resource) you should not read directly a file. At best you should implement retry logic to make sure you bypass any transient errors (read about Transient errors here, here and there or just search the Internet about the term "Transient Error").

In your case I suggest that you wrap your calls aroud the CloudBlockBlob.DownloadToStream method. Thus, you can still use the Stream to parse the file, but you will also work behind the security of the .NET Azure Blob API library which takes care of all transient errors on your behalf.

Your code would look something like this:

// get the CloudblockBlob object
using(MemoryStream blobStream = new MemoryStream())
{
    blobObject.DownloadToStream(blobStream);
    using (var csv = new CsvReader(new StreamReader(blobStream)))
    {
        while (csv.Read())
        {
            try
            {
             //some reading operation
            }
            catch (Exception ex)
            {
                _logger.Error(ex);
            }
        }
        _logger.InfoFormat("Finished {0} reading data #{1}");
    }
}
3
  • I understand that, and have implemented this as a temp solution. Though the answer I'm looking for, is why this happens ? Why does azure of (WCF) think it can close the connection ? And is there a way (or a streamReader implementation) that would handle such kind of error, out of the box ?
    – Marty
    Mar 12, 2014 at 22:02
  • Like - read, if broken, save the position, and reopen the connection, seek to the last point, and continue reading
    – Marty
    Mar 12, 2014 at 22:03
  • I have provided 3 links to understand what is Transient error and why they happen. Try to read the answer, read the references and then ask new questions ...
    – astaykov
    Mar 13, 2014 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.