4

I need to write a c# service ( could be a windows service or a console app) that needs to process large amounts of data ( 100 000 records) stored in a database. Processing each record is also a fairly complex operation. I need to perform a lot of inserts an updates as part of the processing.

We are using NHibernate as the ORM.

One way is to load all the records and process them sequentially... which could turn out to be quite slow. I was looking at multi threading options and was thinking of having multiples threads processing chunks of records simultaneously .

Could anyone give me some pointers on how I should approach this.. considering that I'm using NHibernate and what are the possible gotchas like deadlock etc

Thanks a lot.

2
  • Where are you storing the data that you have processed? 100,000 seperate update statements will kill your database.
    – M Afifi
    Apr 17, 2012 at 14:02
  • any solution with full source code sample ?
    – Kiquenet
    Aug 22, 2012 at 19:54

4 Answers 4

2

You should consider Task Parallel Library.

0
2

Assuming you are using .NET 4.0, you can use the Task Parallel Library (as has been mentioned) to do something like this:

Parallel.ForEach(sourceCollection, item => Process(item));

Your source collection would be an IEnumerable of the loaded records. The library will handle everything for you:

The source collection is partitioned and the work is scheduled on multiple threads based on the system environment. The more processors on the system, the faster the parallel method runs.

It may help to read a tutorial on using Parallel.ForEach(). Also, be aware of potential pitfalls.

1
  • Thanks Jon and all you guys for your suggestions. I'm just looking at the task parallel library. I'm wondering how I would manage NHibernate sessions here as I have no control over the parallel threads . Ideally I would want each parallel thread to have its own session .. any thoughts on how that can be accomplished with TPL ?
    – Sennin
    Mar 17, 2011 at 5:58
0

Sounds like PLINQ is the best solution (Chapter 5 in this article). But as each calculation is working a lot with database, you should create separate session for each thread.

0

Use IStatelessSessions if possible and experiment with the adonet.batch_size property.

Also how performant does it need to be? I'm a fan of NH but this is one scenario where stored procedures might be better

1
  • 1
    High performance is required as always :).. however I'm not too keen on moving all the business logic to a sproc
    – Sennin
    Mar 17, 2011 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.