34

I need to update a very large table periodically and SQLBulkCopy is perfect for that, only that I have a 2-columns index that prevents duplicates. Is there a way to use SQLBulkCopy as "insert or update if exists"?

If not, what is the most efficient way of doing so? Again, I am talking about a table with millions of records.

Thank you

6 Answers 6

21

I published a nuget package (SqlBulkTools) to solve this problem.

Here's a code example that would achieve a bulk upsert.

var bulk = new BulkOperations();
var books = GetBooks();

using (TransactionScope trans = new TransactionScope())
{
    using (SqlConnection conn = new SqlConnection(ConfigurationManager
    .ConnectionStrings["SqlBulkToolsTest"].ConnectionString))
    {
        bulk.Setup<Book>()
            .ForCollection(books)
            .WithTable("Books")
            .AddAllColumns()
            .BulkInsertOrUpdate()
            .MatchTargetOn(x => x.ISBN)
            .Commit(conn);
    }

    trans.Complete();
}

For very large tables, there are options to add table locks and temporarily disable non-clustered indexes. See SqlBulkTools Documentation for more examples.

5
  • 2
    I have to say, this library is chocked full of goodness. I had to write a windows service that did some ETL work and the database interaction was not enough to justify even a MicroORM and this library SqlBulkTools was perfect. Thank you!. Nice work!
    – Hallmanac
    Apr 19, 2017 at 23:21
  • 3
    The documentation link is dead (404). May 11, 2017 at 20:17
  • 6
    SqlBulkTools was sold to ZZZ Projects. May 12, 2017 at 1:54
  • It's 2020 and I am using this package. The performance is great and it's exactly what I needed for bulk insert or update!
    – Rob L
    Apr 24, 2020 at 20:37
  • Also working great, any idea on using a CancellationToken to bail out partial operation (since these will likely be very long running)?
    – morleyc
    Jul 11, 2020 at 16:46
14

I would bulk load data into a temporary staging table, then do an upsert into the final table. See here for an example of doing an upsert.

2
  • 3
    I am not too familiar with SSIS, how exactly will it help accomplishing this task?
    – Sol
    Feb 3, 2011 at 17:34
  • 1
    This is very pricey business. Creating a temporary table and then merging.
    – Alex M
    Nov 17, 2020 at 15:44
12

Not in one step, but in SQL Server 2008, you could:

  • bulk load into staging table
  • apply a MERGE statement to update/insert into your real table

Read more about the MERGE statement

0
11

Instead of create a new temporary table, which BTW consume more space and memory.

I created a Trigger with INSTEAD OF INSERT and use inside MERGE statement.

But don't forget add the parameter SqlBulkCopyOptions.FireTriggers in the SqlBulkCopy.

This is my two cents.

2

Another alternative would be to not use a temporary table but use a stored procedure with a table valued parameter. Pass a datatable to the sp and do the merge there.

2

Got a hint from @Ivan. For those who might need, here's what I did.

create trigger yourschma.Tr_your_triger_name
    on yourschma.yourtable
    instead of INSERT
    as
    merge into yourschma.yourtable as target
    using inserted as source
    on (target.yourtableID = source.yourtableID)
    when matched then
        update
        set target.ID     = source.ID,
            target.some_column = source.some_column,
            target.Amount                       = source.Amount
    when not matched by target then
        insert (some_column, Amount)
        values (source.some_column, source.Amount);
go

Not the answer you're looking for? Browse other questions tagged or ask your own question.