63

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

private void ReadTxtFile()
{            
    string filePath = string.Empty;
    filePath = openFileDialog1.FileName;
    if (string.IsNullOrEmpty(filePath))
    {
        using (StreamReader sr = new StreamReader(filePath))
        {
            String line;
            while ((line = sr.ReadLine()) != null)
            {
                FormatData(line);                        
            }
        }
    }
}

In FormatData() I check the starting word of line which must be matched with a word and based on that increment an integer variable.

void FormatData(string line)
{
    if (line.StartWith(word))
    {
        globalIntVariable++;
    }
}
5
  • You may want to post FormatData (or a simplified version), just in case. Nov 25, 2010 at 5:19
  • @Matthew: just ignore FormatData(), actually whole process is slow, so for troubleshooting i have commented it. Nov 25, 2010 at 5:22
  • You can't ignore the FormatData if you want a fast solution, you be best formatting the data in a separate thread from that reading the data.
    – cspolton
    Nov 25, 2010 at 7:09
  • You've not given much context of how you're accessing globalIntVariable. Given the implementation of FormatData, is it important that the lines are indeed read in order? If not reading multiple larger chunks of data and concurrently aggregating the global variable will be more efficient.
    – cspolton
    Nov 25, 2010 at 9:11
  • You should post actual benchmark data for solutions you have already tried. Nov 26, 2010 at 4:48

9 Answers 9

53

If you are using .NET 4.0, try MemoryMappedFile which is a designed class for this scenario.

You can use StreamReader.ReadLine otherwise.

9
  • 45
    If you're only doing sequential reading you're better of using StreamReader than MemoryMappedFile since it's much faster. Memory mapping is better for random access.
    – Homde
    Nov 25, 2010 at 5:23
  • 3
    Furthermore you probably can't create a ViewAccesor spanning the entire 1 gb so you have to manage that as well as parsing out the linebreaks. FileStreams are 10 times as fast as Memory-Mapped files for sequential reading.
    – Homde
    Nov 25, 2010 at 6:45
  • 3
    @ konrad - agreed, great comment, FYI there is a bit of a discussion of this in O'Reilly's excellent "C# 4.0 in a Nutshell", page 569. For sequential I/O and a 1GB file size, then MemoryMappedFiles are definitely overkill and may slow things down. Aug 23, 2011 at 6:48
  • @TimSchmelter do you really expect to load 1 gb file to memory ?memorymappedfile has a lot of usages... i dont think this is one of them...
    – Royi Namir
    May 15, 2012 at 13:33
  • 1
    @dodgy_coder: I'd be cautitous with such generalizations. Although i agree with your last sentence, but you should better measure it yourself. May 16, 2012 at 7:01
31

Using StreamReader is probably the way to since you don't want the whole file in memory at once. MemoryMappedFile is more for random access than sequential reading (it's ten times as fast for sequential reading and memory mapping is ten times as fast for random access).

You might also try creating your streamreader from a filestream with FileOptions set to SequentialScan (see FileOptions Enumeration), but I doubt it will make much of a difference.

There are however ways to make your example more effective, since you do your formatting in the same loop as reading. You're wasting clockcycles, so if you want even more performance, it would be better with a multithreaded asynchronous solution where one thread reads data and another formats it as it becomes available. Checkout BlockingColletion that might fit your needs:

Blocking Collection and the Producer-Consumer Problem

If you want the fastest possible performance, in my experience the only way is to read in as large a chunk of binary data sequentially and deserialize it into text in parallel, but the code starts to get complicated at that point.

2
  • 1
    +1 The limiting factor is going to be the speed of the reads from disk, so to improve performance have different threads reading vs processing the lines.
    – cspolton
    Nov 25, 2010 at 7:00
  • @Homde the last part you wrote is actually what StreamReader does internally, so why bother?
    – Ozkan
    May 8, 2020 at 14:24
18

You can use LINQ:

int result = File.ReadLines(filePath).Count(line => line.StartsWith(word));

File.ReadLines returns an IEnumerable<String> that lazily reads each line from the file without loading the whole file into memory.

Enumerable.Count counts the lines that start with the word.

If you are calling this from an UI thread, use a BackgroundWorker.

10

Probably to read it line by line.

You should rather not try to force it into memory by reading to end and then processing.

8

StreamReader.ReadLine should work fine. Let the framework choose the buffering, unless you know by profiling you can do better.

5
  • 1
    StreamReader.ReadLine is fine for small file but when i tried it for large file then it is very slow some time not responding. Nov 25, 2010 at 4:29
  • @Mathew: posted code look at it, lines length are not fix some time line contain only 200 word and some time it will be 2000 or greater that it. Nov 25, 2010 at 5:10
  • 2000 isn't a huge amount. That's only 20 KB, if we're talking English words. However, you still may want to call the FileStream constructor manually, specifying the buffer size. I also think FormatData may actually be the issue. That method doesn't keep all the data in memory, does it? Nov 25, 2010 at 5:17
  • @Matthew: i have commented FormatData() and it is still slow, no much significant difference with and without FormatData(). Nov 25, 2010 at 6:03
  • 2
    @Jeevan can you define "slow"? If you read [small file] in n time, then big file will be read in n * [big file]/[small file]. Maybe you are experiencing what's expected?
    – Audrius
    Nov 25, 2010 at 8:53
6

TextReader.ReadLine()

1

I was facing same problem in our production server at Agenty where we see large files (sometimes 10-25 gb (\t) tab delimited txt files). And after lots of testing and research I found the best way to read large files in small chunks with for/foreach loop and setting offset and limit logic with File.ReadLines().

int TotalRows = File.ReadLines(Path).Count(); // Count the number of rows in file with lazy load
int Limit = 100000; // 100000 rows per batch
for (int Offset = 0; Offset < TotalRows; Offset += Limit)
{
  var table = Path.FileToTable(heading: true, delimiter: '\t', offset : Offset, limit: Limit);

 // Do all your processing here and with limit and offset and save to drive in append mode
 // The append mode will write the output in same file for each processed batch.

  table.TableToFile(@"C:\output.txt");
}

See the complete code in my Github library : https://github.com/Agenty/FileReader/

Full Disclosure - I work for Agenty, the company who owned this library and website

1

My file is over 13 GB:

enter image description here

You can use my class:

public static void Read(int length)
    {
        StringBuilder resultAsString = new StringBuilder();

        using (MemoryMappedFile memoryMappedFile = MemoryMappedFile.CreateFromFile(@"D:\_Profession\Projects\Parto\HotelDataManagement\_Document\Expedia_Rapid.jsonl\Expedia_Rapi.json"))
        using (MemoryMappedViewStream memoryMappedViewStream = memoryMappedFile.CreateViewStream(0, length))
        {
            for (int i = 0; i < length; i++)
            {
                //Reads a byte from a stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream.
                int result = memoryMappedViewStream.ReadByte();

                if (result == -1)
                {
                    break;
                }

                char letter = (char)result;

                resultAsString.Append(letter);
            }
        }
    }

This code will read text of file from start to the length that you pass to the method Read(int length) and fill the resultAsString variable.

It will return the bellow text:

1
  • 9
    "It will return the bellow text:" What text?
    – Ctrl S
    Mar 19, 2019 at 19:50
0

I'd read the file 10,000 bytes at a time. Then I'd analyse those 10,000 bytes and chop them into lines and feed them to the FormatData function.

Bonus points for splitting the reading and line analysation on multiple threads.

I'd definitely use a StringBuilder to collect all strings and might build a string buffer to keep about 100 strings in memory all the time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.