4

I'm trying to add to a DataTable some information in Parallel but if the the loop is to long it freezes or just takes a lot of time, more time then an usual for loop, this is my code for the Parallel.For loop:

Parallel.For(1, linii.Length, index =>
                 {
                     DataRow drRow = dtResult.NewRow();
                     alResult = CSVParser(linii[index], txtDelimiter, txtQualifier);

                     for (int i = 0; i < alResult.Count; i++)
                     {
                         drRow[i] = alResult[i];
                     }
                     dtResult.Rows.Add(drRow);
                 }
             );

What's wrong? this Parallel.For loop takes much more time than a normal one, what is wrong?

Thanks!

6
  • 4
    You need to lock the data table. Adding rows is not thread safe Aug 9, 2012 at 12:55
  • so I need first to lock the DataTable, then add a row to it ?
    – XandrUu
    Aug 9, 2012 at 12:56
  • why do you need it parallel in the first place. Aug 9, 2012 at 12:56
  • Yes, every time you do something to shared memory you need to lock it. Aug 9, 2012 at 12:56
  • 1
    @XandrUu if you are dealing with huge files, your biggest problem is not the Parallel - it is the linii[index]. Loading all the lines at once is a killer for performance. Aug 9, 2012 at 13:12

2 Answers 2

5

You can't mutate a DataTable from 2 different threads; it will error. DataTable makes no attempt to be thread-safe. So: don't do that. Just do this from one thread. Most likely you are limited by IO, so you should just do it on a single thread as a stream. It looks like you're processing text data. You seem to have a string[] for lines, perhaps File.ReadAllLines() ? Well, that is very bad here:

  1. it forces it all to load into memory
  2. you have to wait for it all to load into memory
  3. CSV is a multi-line format; it is not guaranteed that 1 line == 1 row

What you should do is use something like the CsvReader from code project, but even if you want to just use one line at a time, use a StreamReader:

using(var file = File.OpenText(path)) {
    string line;
    while((line = file.ReadLine()) != null) {
        // process this line
        alResult = CSVParser(line, txtDelimiter, txtQualifier);

        for (int i = 0; i < alResult.Count; i++)
        {
            drRow[i] = alResult[i];
        }
        dtResult.Rows.Add(drRow);
    }
}

This will not be faster using Parallel, so I have not attempted to do so. IO is your bottleneck here. Locking would be an option, but it isn't going to help you massively.

As an unrelated aside, I notice that alResult is not declared inside the loop. That means that in your original code alResult is a captured variable that is shared between all the loop iterations - which means you are already overwriting each row horribly.


Edit: illustration of why Parallel is not relevant for reading 1,000,000 lines from a file:

Approach 1: use ReadAllLines to load the lines, then use Parallel to process them; this costs [fixed time] for the physical file IO, and then we parallelise. The CPU work is minimal, and we've basically spent [fixed time]. However, we've added a lot of threading overhead and memory overhead, and we couldn't even start until all the file was loaded.

Approach 2: use a streaming API; read each one line by line - processing each line and adding it. The cost here is basically again: [fixed time] for the actual IO bandwidth to load the file. But; we now have no threading overhead, no sync conflicts, no huge memory to allocate, and we start filling the table right away.

Approach 3: If you really wanted, a third approach would be a reader/writer queue, with one dedicated thread processing file IO and enqueueing the lines, and a second that does the DataTable. Frankly, it is a lot more moving parts, and the second thread will spend 95% of its time waiting for data from the file; stick to Approach 2!

5
  • but if i have an text that has 1 000 000 lines i have to wait for all to load, I can do it in Parallel.
    – XandrUu
    Aug 9, 2012 at 13:09
  • Approach 2 was my first methode but I thought that it can be done faster, thanks a lot.
    – XandrUu
    Aug 9, 2012 at 13:21
  • @XandrUu you can't do it faster than the bandwidth-time it takes to do the IO. Aug 9, 2012 at 13:24
  • I'll keep that in mind, thanks a lot :) can you give my a contact of sort like an IM contact, if I have some more questions like this, I liked a lot the detailed explications..
    – XandrUu
    Aug 9, 2012 at 13:26
  • 1
    @XandrUu You should ask another question on SO if you have another problem. That's the purpose of the site after all.
    – Servy
    Aug 9, 2012 at 13:57
1
Parallel.For(1, linii.Length, index =>
{
  alResult = CSVParser(linii[index], txtDelimiter, txtQualifier);

  lock (dtResult)
  {
    DataRow drRow = dtResult.NewRow();
    for (int i = 0; i < alResult.Count; i++)
    {
       drRow[i] = alResult[i];
    }
    dtResult.Rows.Add(drRow);
  }
});
2
  • One side-effect of this is that it randomises the row order. Not sure that is avoidable if the OP wants to use Parallel, though. Aug 9, 2012 at 13:06
  • You can use a shared index, increment it with Interlocked, put all items in a ConcurrentBag with the index, sort and then insert to the datatable (after thread have finished) Aug 9, 2012 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.