11

How can I delete the duplicate records from a Snowflake table?

ID Name
1  Apple
1  Apple
2  Apple
3  Orange
3  Orange

Result should be:

ID Name
1  Apple
2  Apple
3  Orange

11 Answers 11

26

Adding here a solution that doesn't recreate the table. This because recreating a table can break a lot of existing configurations and history.

Instead we are going to delete only the duplicate rows and insert a single copy of each, within a transaction:


-- find all duplicates
create or replace transient table duplicate_holder as (
    select $1, $2, $3
    from some_table
    group by 1,2,3
    having count(*)>1
);

-- time to use a transaction to insert and delete
begin transaction;

-- delete duplicates
delete from some_table a
using duplicate_holder b
where (a.$1,a.$2,a.$3)=(b.$1,b.$2,b.$3);

-- insert single copy
insert into some_table
select * 
from duplicate_holder;

-- we are done
commit;

Advantages:

  • Doesn't recreate the table
  • Doesn't modify the original table
  • Only deletes and inserts duplicated rows (good for time travel storage costs, avoids unnecessary reclustering)
  • All in a transaction
6
  • Thanks for this - I am using a variant, create duplicate_holder as ( select * ... qualify (row_number() over (partition by some_key)) = 2 . Advantage is the ability to write a select * which is handy for wide tables.
    – Jarrad
    May 18, 2021 at 1:36
  • I liked this solution but it misses the ability to look for duplicates using specific PK fields as reference. I have modified the solution to take care of it. Check it out! Mar 17, 2022 at 16:35
  • @Jarrad could you provide a code snippet of your code for handling wide tables?
    – MoneyBall
    Jan 25 at 1:49
  • 1
    Don't forget to drop duplicate_holder once you've verified the result.
    – nofinator
    Feb 9 at 20:38
  • 1
    ahh, I could have used create or replace temp table with temp for automatic deletion Feb 10 at 0:29
13

Here's a very simple approach that doesn't need any temporary tables. It will work very nicely for small tables, but might not be the best approach for large tables.

insert overwrite into some_table
select distinct * from some_table
;

The OVERWRITE keyword means that the table will be truncated before the insert takes place.

2
  • I'm surprised this isn't rated higher since it works perfectly with the least amount of code. Thanks for the tip. Mar 3 at 20:51
  • I really like this -- and to extend it by de-duplicating by primary keys (which is my case, the records aren't identical as we have a load time column), it's easy to add a qualify with row-number combo into the SELECT
    – Bilbottom
    Mar 24 at 8:28
9

If you have some primary key as such:

CREATE TABLE fruit (key number, id number, name text);

insert into fruit values (1,1, 'Apple'), (2,1,'Apple'),
      (3,2, 'Apple'), (4,3, 'Orange'), (5,3, 'Orange');

as then

DELETE FROM fruit
WHERE key in (
  SELECT key 
  FROM (
      SELECT key
          ,ROW_NUMBER() OVER (PARTITION BY id, name ORDER BY key) AS rn
      FROM fruit
  )
  WHERE rn > 1
);

But if you do not have a unique key then you cannot delete that way. At which point a

CREATE TABLE new_table_name AS
SELECT id, name FROM (
    SELECT id
        ,name
        ,ROW_NUMBER() OVER (PARTITION BY id, name) AS rn
    FROM table_name
)
WHERE rn > 1

and then swap them

ALTER TABLE table_name SWAP WITH new_table_name
5
  • Combining WITH ... AS and DELETE throws and error for me, SQL compilation error: syntax error line 10 at position 0 unexpected 'DELETE'.. I think you can only use SELECT, see docs.snowflake.net/manuals/sql-reference/constructs/…
    – ynux
    Nov 7, 2019 at 21:18
  • quite fair point, I've not tested it, but given the CTE is not common (used more than once) it can just be pushed into a sub-select with a WHERE key IN (SELECT...) form Nov 7, 2019 at 21:34
  • very true, replaced with a subselect. Nov 7, 2019 at 21:43
  • Why ALLOW_DUPLICATE is just for JSON file-format and not for all the other file-formats? @SimeonPilgrim
    – Vishrant
    Jun 23, 2020 at 0:35
  • @Vishrant it appears that comment is a unrelated question, that perhaps would be best suited to a new question?? Jun 24, 2020 at 1:17
7

Snowflake does not have effective primary keys, their use is primarily with ERD tools. Snowflake does not have something like a ROWID either, so there is no way to identify duplicates for deletion.

It is possible to temporarily add a "is_duplicate" column, eg. numbering all the duplicates with the ROW_NUMBER() function, and then delete all records with "is_duplicate" > 1 and finally delete the utility column.

Another way is to create a duplicate table and swap, as others have suggested. However, constraints and grants must be kept. One way to do this is:

CREATE TABLE new_table LIKE old_table COPY GRANTS;
INSERT INTO new_table SELECT DISTINCT * FROM old_table;
ALTER TABLE old_table SWAP WITH new_table;

The code above removes exact duplicates. If you want to end up with a row for each "PK" you need to include logic to select which copy you want to keep.

This illustrates the importance to add update timestamp columns in a Snowflake Data Warehouse.

2
  • In my experience duplicate deletion is mostly done manually so swapping the table and then setting the permissions is the easiest.
    – Simon D
    Oct 7, 2019 at 9:31
  • If you want to prevent duplicates, use a merge() instead of an insert(), this will force an update to an existing key instead of adding a duplicate record.
    – Pat
    Sep 16, 2020 at 0:12
6

this has been bothering me for some time as well. As snowflake has added support for qualify you can now create a dedupped table with a single statement without subselects:

CREATE TABLE fruit (id number, nam text);
insert into fruit values (1, 'Apple'), (1,'Apple'),
      (2, 'Apple'), (3, 'Orange'), (3, 'Orange');


CREATE OR REPLACE TABLE fruit AS 
SELECT * FROM 
fruit 
qualify row_number() OVER (PARTITION BY id, nam ORDER BY id, nam) = 1;
SELECT * FROM fruit;

Of course you are left with a new table and loose table history, primary keys, foreign keys and such.

3

Based on above ideas.....following query worked perfectly in my case.

CREATE OR REPLACE TABLE SCHEMA.table
 AS
SELECT
    DISTINCT * 
FROM
    SCHEMA.table
  ;
1

Your question boils down to: How can I delete one of two perfectly identical rows? . You can't. You can only do a DELETE FROM fruit where ID = 1 and Name = 'Apple';, then both rows will go away. Or you don't, and keep both.

For some databases, there are workarounds using internal rows, but there isn't any in snowflake, see https://support.snowflake.net/s/question/0D50Z00008FQyGqSAL/is-there-an-internalmetadata-unique-rowid-in-snowflake-that-i-can-reference . You cannot limit deletes, either, so your only option is to create a new table and swap.


Additional Note on Hans Henrik Eriksen's remark on the importance of update timestamps: This is a real help when the duplicates where added later. If, for example, you want to keep the newer values, you can then do this:

-- setup
create table fruit (ID Integer, Name VARCHAR(16777216), "UPDATED_AT" TIMESTAMP_NTZ);
insert into fruit values (1, 'Apple', CURRENT_TIMESTAMP::timestamp_ntz)
, (2, 'Apple', CURRENT_TIMESTAMP::timestamp_ntz)
, (3, 'Orange', CURRENT_TIMESTAMP::timestamp_ntz);
-- wait > 1 nanosecond
insert into fruit values (1, 'Apple', CURRENT_TIMESTAMP::timestamp_ntz)
, (3, 'Orange', CURRENT_TIMESTAMP::timestamp_ntz);

-- delete older duplicates (DESC)
DELETE FROM fruit
  WHERE (ID
  , UPDATED_AT) IN (
     SELECT ID
     , UPDATED_AT
     FROM (
         SELECT ID
         , UPDATED_AT
         , ROW_NUMBER() OVER (PARTITION BY ID ORDER BY UPDATED_AT DESC) AS rn
         FROM fruit
     )
     WHERE rn > 1
  );
2
  • If the rows are identical then why not use that same trick, instead partition and order on just the ID field. Exactly which row is deleted might be non-deterministic but then they are identical so it doesn't matter
    – Davos
    Jul 10, 2020 at 15:28
  • Another why to think about DELETE is that DELETE is just rewriting the table, thus the CREATE OR REPLACE TABLE AS SELECT DISTINCT is of equal work in snowflake, as it reads all rows, and write all rows, except for the edge case where you only have a small subsection of the data being impacted, that a delete would have pruned (assuming some constriants) Jan 4, 2022 at 0:16
1

simple UNION eliminate duplicates on use case of just all columns/no pks.

anyway problem should he solved as early on ingestion pipeline, and/or use scd etc.

Just a raw magic best way how to delete is wrong in principle, use scd with high resolution timestamp, solves any problem.

you want fix massive dups load ? then add column like batch id and remove all batch loaded records

Its like being healthy, you have 2 approaches:

  1. eat a lot > get far > go-to a gym to burn it
  2. eat well > have healthy life style and no need for gym.

So before discussing best gym, try change life style.

hope this helps, learn to do pressure upstream on data producers instead of living like jesus christ trying to clean up the mess of everyone.

0

The following solution is effective if you are looking at one or few columns as primary key references for the table.

-- Create a temp table to hold our duplicates (only second occurrence)
CREATE OR REPLACE TRANSIENT TABLE temp_table AS (
  SELECT [col1], [col2], .. [coln]
  FROM (
    SELECT *, ROW_NUMBER () OVER(
      PARTITION BY [pk]1, [pk]2, .. [pk]m
      ORDER BY [pk]1, [pk]2, .. [pk]m) AS duplicate_count
      FROM [schema].[table]
  ) WHERE duplicate_count = 2
);


-- Delete all the duplicate records from the table
DELETE FROM [schema].[table] t1
USING temp_table t2
WHERE 
  t1.[pk]1 = t2.[pk]1 AND 
  t1.[pk]2 = t2.[pk]2 AND
  ..
  t1.[pk]n = t2.[pk]m;

-- Insert single copy using the temp_table in the original table
INSERT INTO [schema].[table]
SELECT * 
FROM temp_table;
0

This is inspired by @Felipe Hoffa's answer:

##create table with dupes and take the max id
create or replace transient table duplicate_holder as (
select max(S.ID) ID, some_field, count(some_field) numberAssets
from some_table S
group by some_field
having count(some_field)>1
)

##join back to the original table on the field excluding the ID in the duplicate table and delete. 
delete from some_table as t
USING duplicate_holder as d
WHERE t.some_field=d.some_field
  and t.id <> d.id
-1

Not sure if people are still interested in this but I've used the below query which is more elegant and seems to have worked

create or replace table {{your_table}} as 
select * from {{your_table}}
qualify row_number() over (partition by {{criteria_columns}} order by 1) = 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.