57

I have a transaction that contains multiple SQL Statements (INSERT, UPDATE and/or DELETES). When executing, I want to ignore Duplicate Error statements and continue onto the next statement. What's the best way of doing that?

12 Answers 12

78

I think you are looking for the IGNORE_DUP_KEY option on your index. Have a look at IGNORE_DUP_KEY ON option documented at http://msdn.microsoft.com/en-us/library/ms186869.aspx which causes duplicate insertion attempts to produce a warning instead of an error.

8
  • 44
    @Jonathan Leffler: IGNORE_DUP_KEY won't cause duplicate data. It causes SQL Server to ignore the duplicate key: not inserting it into the database.
    – Ian Boyd
    Feb 3, 2010 at 15:59
  • 9
    @Jonathan Leffler: It's not your fault for mis-understanding the feature. It's is horribly documented: "If you create a unique index, you can set this option to ensure each value in an indexed column is unique.". For years i thought the option was the difference between an error being raised right when you insert, or delayed when you call COMMIT. Turns out it's neither of those. It is better to be named "Ignore, skip, and do not insert, duplicate rows"
    – Ian Boyd
    Feb 3, 2010 at 18:14
  • 2
    @Ian Boyd & Canoehead thanks, indeed it's not well documented and many don't even know it existed let alone how it worked :) May 7, 2012 at 15:42
  • 4
    A have a many-to-many relation, products and categories. I have a table with just product_id and category_id and an unique index. With IGNORE_DUP_KEY i can just add products(a,b,c) to categoies(x,y,x) without worrying about if some of the products are already in one of the categories. I just care about the final result. Apr 22, 2015 at 10:23
  • 2
    @LeifNeland - I agree. Too many people are quick to dismiss this as a bad idea in all cases. There are always legitimate cases where it's not at all unwise to just skip duplicate inserts. Cases that would require more than the character limit of a comment to explain clearly...
    – AaronHS
    Jun 15, 2016 at 23:52
21

Expanding on your comment to SquareCog's reply, you could do:

INSERT INTO X VALUES(Y,Z)    WHERE Y  NOT IN (SELECT Y FROM X)
INSERT INTO X2 VALUES(Y2,Z2) WHERE Y2 NOT IN (SELECT Y FROM X2)
INSERT INTO X3 VALUES(Y3,Z3) WHERE Y3 NOT IN (SELECT Y FROM X3)

Here, I assume that column Y is present in all three tables. Note that performance will be poor if the tables are not indexed on Y.

Oh yeah, Y has a unique constraint on it--so they're indexed, and this should perform optimally.

9
  • 2
    I like this - our deployment scripts are designed to be run many times safely, so INSERT's always have criteria to avoid collisions.
    – n8wrl
    Jul 16, 2009 at 18:06
  • 3
    I tried this approach but my MSSQL Server (2008 R2) comaplains about "Incorrect syntax near the keyword 'WHERE'" - it doesn't seem to like WHERE following INSERT. Are you sure your solution is correct?
    – Dai
    Jun 2, 2012 at 19:16
  • Yes, I am. Post your code, preferrably as a new question, and someone will help you figure out the problem. Jun 4, 2012 at 14:09
  • Is it not possible for you to still get duplicate key errors because you haven’t specified WITH (HOLDLOCK, UPDLOCK) (example)?
    – binki
    Sep 24, 2013 at 19:09
  • 2
    @Dai I had the same issue when using SQL Azure. I am not sure if this is a limitation of SQL Azure or not. This is what I ended up using instead: INSERT into test (col1, col2) SELECT DISTINCT 1, 'a' FROM test WHERE NOT EXISTS (SELECT col1 FROM test WHERE col1 = 1)
    – Cotega
    Mar 21, 2014 at 22:20
14

Although my emphatic advice to you is to structure your sql so as to not attempt duplicate inserts (Philip Kelley's snippet is probably what you need), I want to mention that an error on a statement doesn't necessarily cause a rollback.

Unless XACT_ABORT is ON, a transaction will not automatically rollback if an error is encountered unless it's severe enough to kill the connection. XACT_ABORT defaults to OFF.

For example, the following sql successfully inserts three values into the table:

create table x ( y int not null primary key )

begin transaction
insert into x(y)
values(1)
insert into x(y)
values(2)
insert into x(y)
values(2)
insert into x(y)
values(3)
commit

Unless you're setting XACT_ABORT, an error is being raised on the client and causing the rollback. If for some horrible reason you can't avoid inserting duplicates, you ought to be able to trap the error on the client and ignore it.

0
12

If by "Ignore Duplicate Error statments", to abort the current statement and continue to the next statement without aborting the trnsaction then just put BEGIN TRY.. END TRY around each statement:

BEGIN TRY
    INSERT ...
END TRY
BEGIN CATCH /*required, but you dont have to do anything */ END CATCH
...
1
  • Unfortunately, we're using SQL Server 2000 right now so I don't think the Try .. Catch statements are available in that version.
    – Kingamoon
    Jul 16, 2009 at 17:45
10

I'd like to chime in with the following:

If 99% of your data is going to insert without error doing a select beforehand results in a huge performance drop (like, in my case, from 200 lines/sec to 20 lines/sec) compared to "dumb" inserts and catching the occasional error.

After ignoring the "Violation of PRIMARY KEY constraint" errors things went back to being bottlenecked by other resources (headroom being defined as "what the bottlenecking resources don't have").

Which is the whole reason I landed on this discussion in the first place.

4

OK. After trying out some error handling, I figured out how to solve the issue I was having.

Here's an example of how to make this work (let me know if there's something I'm missing) :

SET XACT_ABORT OFF ; -- > really important to set that to OFF
BEGIN
DECLARE @Any_error int
DECLARE @SSQL varchar(4000)
BEGIN TRANSACTION
    INSERT INTO Table1(Value1) VALUES('Value1')
    SELECT @Any_error = @@ERROR
    IF @Any_error<> 0 AND @Any_error<>2627 GOTO ErrorHandler

    INSERT INTO Table1(Value1) VALUES('Value1')
    SELECT @Any_error = @@ERROR
    IF @Any_error<> 0 AND @Any_error<>2627 GOTO ErrorHandler

    INSERT INTO Table1(Value1) VALUES('Value2')
    SELECT @Any_error = @@ERROR
    IF @Any_error<> 0 AND @Any_error<>2627 GOTO ErrorHandler

    ErrorHandler: 
       IF @Any_error = 0 OR @Any_error=2627
       BEGIN 
           PRINT @ssql 
           COMMIT TRAN
       END
       ELSE 
       BEGIN 
           PRINT @ssql 
           ROLLBACK TRAN 
       END
END

As a result of the above Transaction, Table1 will have the following values Value1, Value2.

2627 is the error code for Duplicate Key by the way.

Thank you all for the prompt reply and helpful suggestions.

2

Use IGNORE_DUP_KEY = OFF during primary key definition to ignore the duplicates while insert. for example

create table X( col1.....)

CONSTRAINT [pk_X] PRIMARY KEY CLUSTERED 
(
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON, FILLFACTOR = 70) ON [PRIMARY]
) ON [PRIMARY]
1
INSERT INTO KeyedTable(KeyField, Otherfield)
SELECT n.* FROM 
    (SELECT 'PossibleDupeLiteral' AS KeyField, 'OtherfieldValue' AS Otherfield
     UNION ALL
     SELECT 'PossibleDupeLiteral', 'OtherfieldValue2'
    )
LEFT JOIN KeyedTable k
    ON k.KeyField=n.KeyField
WHERE k.KeyField IS NULL

This tells the Server to look for the same data (hopefully the same speedy way it does to check for duplicate keys) and insert nothing if it finds it.

I like the IGNORE_DUP_KEY solution too, but then anyone who relies on errors to catch problems will be mystified when the server silently ignores their dupe-key errors.

The reason I choose this over Philip Kelley's solution is that you can provide several rows of data and only have the missing ones actually get in:

1

I came here because I was trying to do the same thing; I knew I had dupes in the source data but only wanted to update the target data and not add the dupes.

I think a MERGE works great here because you can UPDATE or DELETE things that are different and INSERT things that are missing.

I ended up doing this and it worked great. I use SSIS to loop through Excel files and load them into a "RAW" SQL table with dupes and all. Then I run a MERGE to merge the "raw" table with the production table. Then I TRUNCATE the "raw" table and move to the next Excel file.

1
  • 1
    If you're loading into this table a lot, why not just clean up the dupes by loading the current table into a new table with a unique index with IGNORE_DUP_KEY = ON. Delete your original table and rename the new table. The you could directly load into the dupe-free table. Seems like it would save a great deal of time in the future for minimal effort.
    – Mark
    Jul 14, 2016 at 19:04
1

For SQL server 2000:

     INSERT INTO t1 (ID, NAME)
      SELECT valueid, valuename
      WHERE  NOT EXISTS
               (SELECT 0
                FROM   t1 as t2
                WHERE  t2.ID = valueid AND t2.name = valuename)
0

Well you could solve this with a temp table..

DECLARE @RoleToAdds TABLE
([RoleID] int, [PageID] int)

INSERT INTO @RoleToAdds ([RoleID], [PageID])
    VALUES
    (1, 2),
    (1, 3),
    (1, 4),
    (2, 5)

INSERT INTO [dbo].[RolePages] ([RoleID], [PageID])
    SELECT rta.[RoleID], rta.[PageID] FROM @RoleToAdds rta WHERE NOT EXISTS 
        (SELECT * FROM [RolePages] rp WHERE rp.PageID = rta.PageID AND rp.RoleID = rta.RoleID)

This might not work for big amounts of data but for a few rows it should work!

-1

Keys must be unique. Don't do that. Redesign as needed.

(if you are trying to insert, then delete, and the insert fails... just do the delete first. Rollback on error in either statement).

1
  • No I don't think I made my question clear. INSERT INTO X VALUES(Y,Z) INSERT INTO X2 VALUES(Y2,Z2) INSERT INTO X3 VALUES(Y3,Z3) Let's say the second statement causes a duplicate error. I want to ignore it and continue execute the 3rd statement. The behavior in a Transaction is, it throws an error and it exists (Rolls back).
    – Kingamoon
    Jul 16, 2009 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.