0

I have some crappy data from a source I cannot control, it needs to go into a table with a composite primary key that looks like this:

PK_Part1, PK_Part2, StringData, DateData

My crappy data has full duplicates, PK Duplicates with different StringData, PK Duplicates with different DateData, and PK Duplicates with different StringData and DateData.

So I might see:

1234,1234,Blah,2011-1-1
1234,1234,Blah,2011-1-1
4321,4321,Blah,2011-1-1
4321,4321,Blah,2011-10-10
5678,5678,Blah,2011-1-1
5678,5678,Blah1,2011-1-1
8765,8765,Blah,2011-1-1
8765,8765,Blah,2011-10-10
8765,8765,Blah1,2011-10-10

How so I clean this up in SQL Server 2008? given that:
A) I want only the data associated with the latest date
B) I'm trying to force the issue with the source about the string data, but for now longer string is better, same length either will do.
C) I have to assume the source will be of no help and load everything now

I had hoped to use MERGE but it seems to compare all rows of the Source table and Target table before doing any of the 'MATCH' or 'NO MATCH' statements so I got PK violations, and removing the PK constraint let all the duplicates in.

5
  • Say you have incoming data that according to your rules is better than the existing, you want to DELETE the existing and INSERT the new one, is that right? Mar 16, 2011 at 19:30
  • btw you're wrong about the MERGE statement - it does the MERGEing at a row-by-row level, so it is probably what you want Mar 16, 2011 at 19:32
  • @Richard aka cyberkiwi: Update is fine - it needs to replace it somehow though, I'm still picking up SQL so I'll happily take advice on any aspect of this process. Thanks!
    – DKnight
    Mar 16, 2011 at 19:35
  • @Richard aka cyberkiwi: RE: Merge - You're sure? I'll check my merge statement but it looked pretty straightforward.
    – DKnight
    Mar 16, 2011 at 19:37
  • If you can first make most of the values equal with e.g. LTRIM(RTRIM(xx)) , make dates really DateTimes and SELECT them with DISTINCT
    – ralf.w.
    Mar 16, 2011 at 20:12

4 Answers 4

4

If you don't have that data in SQL Server already: BULK INSERT that into a temporary table:

CREATE TABLE #tempStaging
(PK_Part1 INT, PK_Part2 INT, StringData VARCHAR(500), DateData DATE)

BULK INSERT #tempStaging
FROM 'c:\yourfile.txt'
WITH (FIELDTERMINATOR =',',
     ROWTERMINATOR ='\n')

Then you should be able to do something like:

;WITH CleaupData AS
(
  SELECT 
      PK_Part1, PK_Part2, StringData, DateData,
      ROW_NUMBER() OVER(PARTIION BY PK_Part1, PK_Part2
                        ORDER BY DateData DESC, LEN(StringData) DESC) as 'RowNum'
  FROM
      #tempStaging
)
INSERT INTO dbo.YourTargetTable(PK_Part1, PK_Part2, StringData, DateData)
    SELECT PK_Part1, PK_Part2, StringData, DateData 
    FROM CleanupData
    WHERE RowNum = 1

This will "partition" your data based on some criteria (some ID or something), and each partition of data is order by date (descending - newest first).

So the entry with the RowNum = 1 is the newest entry for each partition - pick that one and toss out all others, and your data is cleaned up !

HINT: this assumes that your target table is empty! If that's not the case, then yes - you might need to apply a MERGE statement instead, based on the CTE that selects out the data to keep from the BULK INSERT.

3
  • 1
    Definitely the way to go. He also has, under 2), something about string length. I'm assuming that if the dates are the same and string A is longer than string B, then take string B. So I'd throw in a ORDER BY LEN(StringData) DESC after your DateColumn sort. But I might be mis-reading.
    – Mike M.
    Mar 16, 2011 at 19:37
  • @Richard aka cyberkiwi: yes, this works assuming you're either BULK INSERTing your data into a staging table without any PK, or you have a table that needs to be scrubbed (and doesn't enforce PK just yet)
    – marc_s
    Mar 16, 2011 at 19:45
  • Thanks so much for this - I could have easily put this out to a temporary table and merged that. I really appreciate the time you put into this
    – DKnight
    Mar 16, 2011 at 20:51
2

The data form the source should go into a temp table, a holding temp area. Then you can choose the best one from that (since your sample data contains duplicate part1+part2 even within the input data)

Sample table and temp table

create table pkdup(
    PK_Part1 int, PK_Part2 int, StringData varchar(100), DateData datetime,
    primary key (PK_Part1,PK_Part2))
insert pkdup select 1234,1234,'', GETDATE()+1000

create table #tmp(col1 nvarchar(max), col2 nvarchar(max), col3 nvarchar(max), col4 datetime)
insert #tmp values
(1234,1234,'Blah','2011-1-1'),
(1234,1234,'Blah','2011-1-1'),
(4321,4321,'Blah','2011-1-1'),
(4321,4321,'Blah','2011-10-10'),
(5678,5678,'Blah','2011-1-1'),
(5678,5678,'Blah1','2011-1-1'),
(8765,8765,'Blah','2011-1-1'),
(8765,8765,'Blah','2011-10-10'),
(8765,8765,'Blah1','2011-10-10');

The merge statement

merge pkdup as target
using (
    select col1, col2, col3, col4
    from (select *, row_number() over (
        partition by col1, col2
        order by col4 desc, len(col3) desc) rownum
        from #tmp) t
    where rownum=1 -- only the best
    ) as source
on source.col1=target.PK_Part1 and source.col2=target.PK_Part2
WHEN MATCHED AND (source.col4 > target.datedata or (source.col4=target.datedata and len(source.col3) > target.stringdata))
    THEN UPDATE SET target.stringdata = source.col3, target.datedata = source.col4
WHEN NOT MATCHED THEN
    INSERT (PK_Part1, PK_Part2, StringData, DateData)
    VALUES (source.col1, source.col2, source.col3, source.col4);
1
  • Thanks! - I had dumbed it down and had to have some caffine before I tackled the real problem but this was exactly what I needed. Now to read up on how partition works.
    – DKnight
    Mar 16, 2011 at 20:50
1

We usually put such data into a staging table and then get rid of the duplicates in the staging table before we try to run a merge statement.

0
0

not sure if you can apply a string length function in a join, but if you can, try this:

select PK_Part1, PK_Part2, max_date, max_len, first(StringData) as first_string
from        
   (select PK_Part1, PK_Part2, max_date, max(len(StringData)) as max_len
    from table inner join
           (select PK_Part1, PK_Part2, max(DateData) as max_date
           from table
           group by
           PK_Part1, PK_Part2) md
    on table.PK_Part1 = md.PK_Part1 and 
           table.PK_Part2 = md.PK_Part2 and 
           table.DateData = md.max_date
    group by
           PK_Part1, PK_Part2, max_date) ml
   inner join table on 
           table.PK_Part1 = ml.PK_Part1 and 
           table.PK_Part2 = ml.PK_Part2 and 
           table.DateData = ml.max_date and
           len(table.StringData) = ml.max_len
   group by
           PK_Part1, PK_Part2, max_date, max_len

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.