2

When it comes down to INSERTing large amount of data (say 100.000 rows) in an ARRAY object, which one do you think it would be faster?

  1. Through Entity Framework? (call SaveChanges() when u have inserted 100.000 rows)
  2. Looping every row (100.000 times) with an INSERT stored procedure?

If you could also provide a reference that would be very awesome.

Thanks

2
  • You can pass data to store procedure XML parameter.
    – Jagz S
    Dec 4, 2012 at 12:25
  • So how did it measure with TVPs via stored procedure? I would expect it to be ~100x faster than 1 or 2 listed above (see answer)
    – bryanmac
    Dec 6, 2012 at 0:32

3 Answers 3

3

Looping 100K times calling a stored procedure will at a minimum create 100K cross process and/or cross network calls will be slow.

If you're using SQL server, another option is to use TVPs (table value paramaters) to avoid calling insert in a loop from your C# code. It allows you to pass a table of data to a stored procedure in one call.

From the link above, they recommend 1000 rows at a time (but always measure and experiment for your app):

Using table-valued parameters is comparable to other ways of using set-based variables; however, using table-valued parameters frequently can be faster for large data sets. Compared to bulk operations that have a greater startup cost than table-valued parameters, table-valued parameters perform well for inserting less than 1000 rows.

So, maybe try out looping 100 times passing a 1000 rows at a time (instead of cross the boundary 100K times).

You might also want to re-evaluate why asp.net has 100K items at one time in your app. Is that passed up to the server and held in memory at once with possible memory issues? Can that be broken up? Are you doing data processing where asp.net is reading out and processing 100K rows where a sql server agent job might be more appropriate? If you provide more details on the data flow of your app and what it's doing, folks might be able to offer more options.

4
  • This method will be run in about every 5 hours or so. Thats why I estimated about 100k items.
    – E-A
    Dec 4, 2012 at 14:04
  • So, how did it measure with 100 loops sending a TVP with 1000 rows? My guess is ~100 times faster than both of your options.
    – bryanmac
    Dec 4, 2012 at 20:56
  • 2
    1) Looping the SP 100k times: 40 sec 2) Using EF to insert 100k items: 19 sec 3) Table value inserting 100k items: 1 sec.. Thank you bryan! This was awesome.
    – E-A
    Dec 10, 2012 at 8:32
  • Cool - make sure the items in a batch are not unbounded. I would break into something like 1000 and loop. It might not be quite as fast but it will prevent a spike in 5 hrs running the AT or SQL out of memory since it passes that set in a table. Good luck.
    – bryanmac
    Dec 11, 2012 at 4:40
3

Actually fastest way is to use SqlBulkCopy object, which is designed exactly for this situation

1

The stored procedure will be faster. Entity Framework does not have batching, so you suffer the performance overhead of EF on top of hitting the database 100k times. Raw ado is going to be faster than an ORM for this kind of stuff.

Here is a link of some comparisons http://blog.staticvoid.co.nz/2012/03/entity-framework-comparative.html It does not have raw ado.net in the comparison but dapper is the closest you will see to speed to that.

4
  • Hmm.. I tested about a minute ago and as avarage here is the results of both with 100.000 rows inserting: 21467 miliseconds with EF, 41133 miliseconds with SP (continues over already OPEN connection, not re-opening or closing during the transaction)
    – E-A
    Dec 4, 2012 at 13:35
  • 1
    Right, 100K in a loop will be slow regardless. TVPs will reduce your most expensive cost - the network 100K times.
    – bryanmac
    Dec 4, 2012 at 13:43
  • I guess it could depend on what you are doing in your sproc, have you tested doing a simple insert as a benchmark? In my own experience raw sql has always been significantly faster. What version of EF are you using? Bryanmac is correct, 100k connections is going to be slow no matter what. Dec 4, 2012 at 14:06
  • I have tried EF first and I'Ll publish my calculations. But never got the chance of trying your way. Today I'Ll do that and than I'll share the results here as an answer. Thank you very much FettMo.
    – E-A
    Dec 7, 2012 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.