0

I have a very large text file that is provided by a customer that I need to read line by line and process through an internal system web based processing system by posting requests (up to 10 threads at a time).

I know how to read the file sequentially, but have no clue how to launch threads (up to max of 10) to process simultaneously. I could process one at a time but the files are huge and that could take forever. Any help would be great, suggestions, or snippets?

3
  • 3
    Have a look at tasks. Post some code trying, explain what goes wrong.
    – BugFinder
    Aug 3, 2015 at 22:05
  • 2
    I am not sure that multithreading will bring performance benefits. Multithread will only be relevant if the process of each line is a time consuming process. Otherwise, your may suffer from hard disk head movements due to concurrent access to the disk.
    – Graffito
    Aug 3, 2015 at 22:14
  • Do you need results from processing before handling the next line?
    – Erik Eidt
    Aug 3, 2015 at 22:34

3 Answers 3

1

If most of your work is post-read processing (so the hard disk access is not the bottleneck as Graffito mentioned) then you could be on the right track. Thread work in the same space is very difficult to get right, though. And without code from you it is hard to make specific suggestions. If you have a number of different methods to call and they do not need to be called sequentially, but they all need to finish before you can move forward, then you just make a task for each of those methods in the calling method, and you use Task.WaitAll to hold everything up until the last task returns.

If on the other hand, you want all of the tasks to work in all of the same methods at the same time, each on a different chunk of text, this can be trickier. In this case, to keep one thread from fiddling with fields while another is using them, there are locks and other tools to make methods more thread safe. But the easiest thing to do if your system can handle it might be to move all of the work you want to multi-thread into a separate class. That's methods, fields, and all. Then you create an instance of that class for each thread. When thread0 or task0 is messing around inside myThreadworkClass[0] and thread1 or task1 is inside myThreadworkClass[1], and so on, they will never meet up and get into fights with one another.

So it depends on the specifics of your needs, but hopefully this post will help get you moving in the right direction. Post code if you need more specific help.

1

You should use only one thread to open the file. File.ReadLines Method reads line-by-line without loading the whole file into memory at once. Then you can use multiple threads to handle data that you read. Parallel.ForEach is an option an available from .net Framework 4.0.

Parallel.ForEach(File.ReadLines("file.txt"), (line, action, position) =>
{
    // your code here
})

Use ParallelOptions.MaxDegreeOfParallelism option to limit number of max threads.

1
  • I found something similar to your suggestion, it seems to work well with no issues so far. string filename = "C:\_In\\Input.txt"; int n = 10; foreach (var line in File.ReadLines(filename).AsParallel().WithDegreeOfParallelism(n)) { BuildAndSendRequest(line); /* this creates an xml request from the line data and posts it to te web site to process*/ } I'd like to enhance the code in the threads to be able to update variables defined in Main that are periodically read to update progress bars and labels. Can I increment those int values or is there some thread safe way I have to do it?
    – JST Matt
    Aug 4, 2015 at 16:06
0

There is a simple advice: Don't.

Read from one thread line by line and you will be the fastest. If you try it several times with one file you will see benefits from using multiple threads. This works because you are not reading from disc but from the file system cache. But as you said you have a very large file which is

  • Not in the file system cache in the first place
  • Does not fit into the file system cache anyway.

As long as you do not read the file from a raid array there is little point to use more than one thread.

Assuming you are reading the data from a spinning hard disc you will loose about a factor two of throughput because if you are reading in parallel from two different places from the hard disc. The disc head needs to move which takes 6-10ms for every seek operation to move to the new read location. If you spare the seeks you will be faster.

This sounds more bad than it really is. In reality your single threaded application is already using async IO without knowing it. If you read from a file with the default file flags (Sequential Scan could be worthwile) the operating system will prefetch the data from the hard disc while you are processing the string. In an optimal situation your processing takes just long enough so the next Read operation completes immediately due to the prefetched data.

You are getting the benefits of async IO without complicating your code. You can get a little faster by using unbuffered IO which spares you one memory buffer copy but to make this work in managed code you need to allocate page aligned memory of the HDD sector size. You need to read that from the HDD. Otherwise your code will fail on computers which e.g. use SSDs or hard drives with 4KiB per sector.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.