4

Based on the idea presented here, I've created a QueuedDataReader which wraps a ConcurrentQueue in an IDataReader so I can pass it to SqlBulkCopy and 'stream' data into the database.

It works ok, and pretty fast, but the log file grows quite dramatically, even though I believe I've implemented everything suggested here, here (and numerous other places).

Two possibly important details are:

  • I'm inserting into a dozen tables in parallel (that is, one Task per table)
  • The tables have IDENTITY columns (which SqlBulkCopy is not inserting into), so I don't think the 'SORT' provisos are relevant.

In addition to that:

  • The tables are empty when I start inserting.
  • I've tried both CLUSTERED and NONCLUSTERED indexes on the PRIMARY KEY, not much observable difference.
  • I'm using SqlBulkCopyOptions.TableLock
  • I've tried with and without UseInternalTransactions, and various BatchSizes.
  • The database is in SIMPLE recovery mode
  • The tables do have FK constraints, but I've tried disabling/re-enabling them and it doesn't stop the log file growing (during insertion)

The question:

  • What are some things I could look at to try and troubleshoot what could be causing the log file growth?
2
  • Look at the log contents using fn_dblog (instructions available on the web). What is the bulk of the operations there? Maybe you can post a screenshot of a representative sample. Did you try no batch size (0)? Because only the first batch finds the table empty.
    – usr
    Oct 28, 2013 at 12:46
  • @usr, currently trying batch size 0, but that also means I have to do timeout 0, which feels... dangerous. I'll look into fn_dblog.
    – Benjol
    Oct 28, 2013 at 12:56

1 Answer 1

0

Whilst writing my question, I discovered another excellent resource here, which pointed to BatchSize as a potential culprit.

The results are quite surprising. If we use the BatchSize parameter, performance get worse and worse as we set it to lower values.

Using a very small value for the batch size the network bandwidth stays well under 20% of usage for the whole process (please note that the 500 batch size graph upper limit is 25%, which is different from the other graphs). Any value below 10.000 decreases performance in a very heavy way, leading to very poor time and a heavy log file usage.

When we reach 10.000 of Batch Size, then the difference in time among the various tests becomes very small. But, since we have 8 threads writing 750.000 rows each, then we only have 75 chunks of data sent from each thread. Needless to say, the best performance is obtained when we use 0 as BatchSize, sending the whole 750.000 rows in a single batch.

It might be interesting to discover if there is any benefit, apart from performance, by using the BatchSize parameter. We did not find anything mentioned in the Books On Line nor have we ever seen anything interesting during our experience, this lead us to say that the best thing to do with BatchSize is to leave it to zero, which is its default value, since any value different from that will decrease the performance of the load process.

I had previously abandoned BatchSize = 0 because of timeout errors, but by setting my BulkCopyTimeout to zero, I get some log file growth, but far less than previously.

UPDATE. After messing around for too long doing manual tests, I finally knuckled down to writing an automated test suite, trying variations of BatchSize(0-5000), enqueuing rates, and clustered/nonclustered indexes. (In my case, I'm writing data to a dozen tables in parallel).

This is what I've found:

  • Tests of 200000 inserts: If primary keys are CLUSTERED or BatchSize is zero, I get no log file growth.
  • Tests of 1 million inserts: even BatchSize=0 has the log file growing (though considerably less than for other BatchSizes). CLUSTERED still prevents log growth, whatever the BatchSize.
  • Outside of those conditions, everything depends on insertion speed. As soon as I 'push' too hard, the log file starts growing.

Note, my tests write data at a given rate into a queue wrapped in an IDataReader, which is dequeued by BulkInsert. One queue/BulkInsert for each table. Next step is to try with bigger data sets.

5
  • It is definitely possible to get minimally-logged bulk inserts with non-empty tables. Not sure what's going wrong in your case because the "checklist" you posted seems quite good. I encourage you to look at the log records to see what is being written there. You should find 99% of them to be allocations.
    – usr
    Oct 29, 2013 at 11:25
  • The output from fn_dblog is rather difficult to interpret (and there's lots of it for a bulk insert), but I observe that there is a lot of allocations, as you say, but a corresponding amount of deallocation. Is that expected? It's also logging the IDENT_NEWVAL, and the insertions to the (unclustered) PK index.
    – Benjol
    Oct 30, 2013 at 8:51
  • Well, I don't know from the top of my head what the right or wrong things would look like, but if I saw the the log contents I could probably make sense of it. Here is the query I always use for that: pastebin.com/YZ215dSY. You can also change it to TOP 100 ORDER BY NEWID() to get a representative sample. Be sure to clear the log first.
    – usr
    Oct 30, 2013 at 9:06
  • @usr, currently tearing my hair out :( Having difficulty finding a sufficiently simplified example which gives repeatable results...
    – Benjol
    Oct 30, 2013 at 13:19
  • If you script out the database as well that should give repeatable results. When I ran your scripts I increased the data amount by 100x again to make sure to trigger "bulk behavior" and to cross any thresholds that might be there.
    – usr
    Oct 30, 2013 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.