0

I need help in constructing a query using Teradata(Version: 16.0+) OLAP functions with below scenario for comparing & eliminate rollup duplicates from a Teradata table.

I have below 9 records in a table ABC.

          Existing Data(Table - ABC)    


            ACCOUNT_ID  EXT_REF_NO  SERIAL_NUM  RECORD_START_DT RECORD_END_DT
        1   100000000002195 8495752450757852    341FE4E6A1AF    8/13/2019 12:24:42  8/20/2019 23:59:59
        2   100000000002195 8495752450757852    342FE4E6A1AF    8/21/2019 08:49:08  8/25/2019 23:59:59
        3   100000000002195 8495752450757852    343FE4E6A1AF    8/27/2019 02:42:46  8/26/2019 23:59:59
        4   100000000002195 8495752450757852    344FE4E6A1AF    8/28/2019 06:33:50  8/28/2019 23:59:59
        5   100000000002195 8495752450757852    345FE4E6A1AF    8/30/2019 02:35:32  8/31/2019 23:59:59
        6   100000000002195 8495752450757852    346FE4E6A1AF    9/2/2019 00:25:05   9/1/2019 23:59:59
        7   100000000002195 8495752450757852    347FE4E6A1AF    9/3/2019 03:33:28   9/3/2019 23:59:59
        8   100000000002195 8495752450757852    348FE4E6A1AF    9/4/2019 18:35:45   9/8/2019 23:59:59
        9   100000000002195 8495752450757852    349FE4E6A1AF    9/10/2019 11:22:54  3/16/2020 23:59:59

Output      

            ACCOUNT_ID  EXT_REF_NO  SERIAL_NUM  RECORD_START_DT RECORD_END_DT           
        1   100000000002195 8495752450757852    341FE4E6A1AF    8/13/2019 12:24:42  8/26/2019 23:59:59
        2   100000000002195 8495752450757852    342FE4E6A1AF    8/28/2019 06:33:50  8/28/2019 23:59:59
        3   100000000002195 8495752450757852    343FE4E6A1AF    8/30/2019 02:35:32  9/1/2019 23:59:59
        4   100000000002195 8495752450757852    345FE4E6A1AF    9/3/2019 03:33:28   9/8/2019 23:59:59
        5   100000000002195 8495752450757852    346FE4E6A1AF    9/10/2019 11:22:54  3/16/2020 23:59:59
  1. RECORD_END_DT should be always greater than RECORD_START_DT

  2. We will only consider those records for roll up where Record_start_dt of current row = (RECORD_END_DT + 1 day) for previous row, if the difference is more than 1 day it will not be considered

  3. You can find violating point 1 for row no - 3 & 6, this was actually a bug for same day expired records while data entry was done, you can virtually consider RECORD_START_DT as 8/26/2019 00:00:00 & 9/2/2019 00:00:00 respectively for row no - 3 & 6 for calculation

  4. ACCOUNT_ID,EXT_REF_NO,SERIAL_NUM all 3 should be considered for partition by

I have tried something like below. Only getting one row output with minimum DEVICE_START_DATE & maximum DEVICE_END_DATE like below:

ACCOUNT_ID EXT_REF SERIAL_NUM DEVICE_START_DATE DEVICE_END_DATE 100000000002195 8495752450757852 341FE4E6A1AF 8/13/2017 12:24:42.000000 9/16/2017 23:59:59.000000

 Query: SELECT 
      ACCOUNT_ID,
      EXT_REF, 
      SERIAL_NUM, 
      CASE WHEN (B.DIFF_DAYS <= 1 OR B.DIFF_DAYS IS NULL) THEN
      min(DEVICE_START_DATE) 
      OVER (PARTITION BY ACCOUNT_ID,EXT_REF,SERIAL_NUM order by 
      DEVICE_END_DATE desc)
      WHEN (B.DIFF_DAYS > 1 ) THEN
      min(DEVICE_START_DATE) 
      OVER (PARTITION BY ACCOUNT_ID,EXT_REF,SERIAL_NUM order by 
      DEVICE_END_DATE desc) 
      END AS DEVICE_START_DATE,
      DEVICE_END_DATE
      FROM
      (SELECT A.ACCOUNT_ID,
      A.EXT_REF, 
      A.SERIAL_NUM, 
      A.DEVICE_START_DATE, 
    A.DEVICE_START_DATE_VIRTUAL,
    A.DEVICE_END_DATE, 
    MIN(A.DEVICE_END_DATE)
    OVER ( PARTITION BY A.ACCOUNT_ID,A.EXT_REF,A.SERIAL_NUM ORDER BY A.DEVICE_END_DATE
    ROWS BETWEEN 1 PRECEDING AND 1 PRECEDING) AS DEVICE_END_DATE_PREVIOUS_ROW,
            TRUNC(A.DEVICE_START_DATE_VIRTUAL) - TRUNC(DEVICE_END_DATE_PREVIOUS_ROW) AS DIFF_DAYS
    FROM
    (SELECT 
    ACCOUNT_ID, 
    EXT_REF, 
    SERIAL_NUM, 
    DEVICE_START_DATE, 
    CASE WHEN DEVICE_START_DATE > DEVICE_END_DATE  
    THEN (DEVICE_START_DATE - INTERVAL '1' DAY)   
    ELSE DEVICE_START_DATE END AS DEVICE_START_DATE_VIRTUAL,
    DEVICE_END_DATE
    FROM NDW_XH_TEMP_TABLES.TEST) A) B
    QUALIFY
    ROW_NUMBER() 
    OVER (PARTITION BY ACCOUNT_ID,EXT_REF,SERIAL_NUM order by DEVICE_END_DATE desc) = 1;
3
  • Those SERIAL_NUMs are unique values, so they can't be considered for calculating dups. Do you want to keep the violating dates or should they be fixed? If you don't care about the time portion could you switch to DATEs instead?
    – dnoeth
    May 19, 2020 at 8:21
  • Hi Dieter, you can consider actual start dates instead of violating start dates virtually(Row no 3 & 6) while writing query, something like ( CASE WHEN RECORD_START_DT > RECORD_END_DT THEN RECORD_START_DT = (RECORD_START_DT - INTERVAL '1' DAY) ). RECORD_START_DT & RECORD_END_DT are timestamp(6) fields in table, ACCOUNT_ID is integer, others are VARCHAR. You can switch to DATE if you want. Thanks. May 19, 2020 at 8:49
  • We cannot modify the source since we don't own the table, but later we can share an one time update to source team to modify such faulty Record start dates. May 19, 2020 at 9:09

1 Answer 1

1

You need nested OLAP-functions, this should work as expected:

SELECT 
   ACCOUNT_ID
  ,EXT_REF_NO
  ,SERIAL_NUM
  ,Coalesce(Lag(next_start)
            Over (PARTITION BY ACCOUNT_ID, EXT_REF_NO
                  ORDER BY next_start NULLS LAST)
           ,min_start) AS RECORD_START_DT

-- If your Teradata version doesn't support LAG/LEAD you must switch to the MAX version
--  ,Coalesce(Max(next_start)
--            Over (PARTITION BY ACCOUNT_ID, EXT_REF_NO
--                  ORDER BY next_start NULLS LAST
--                  ROWS BETWEEN 1 Preceding AND 1 Preceding)
--           ,min_start) AS RECORD_START_DT
  ,RECORD_END_DT  
FROM
 (
   SELECT
      ACCOUNT_ID
     ,EXT_REF_NO
     ,SERIAL_NUM
     ,RECORD_START_DT
     ,RECORD_END_DT

     -- to check for a gap
     ,Lag(fixed_start)
      Over (PARTITION BY ACCOUNT_ID, EXT_REF_NO
            ORDER BY fixed_start DESC) AS next_start
--     ,Max(fixed_start)
--      Over (PARTITION BY ACCOUNT_ID, EXT_REF_NO
--            ORDER BY fixed_start DESC
--            ROWS BETWEEN 1 Preceding AND 1 Preceding) AS next_start

     -- used in the outer COALESCE to get the min start for the 1st group
     ,Min(RECORD_START_DT)
      Over (PARTITION BY ACCOUNT_ID, EXT_REF_NO) AS min_start

     -- gap detection
     ,CASE WHEN Cast(RECORD_END_DT AS DATE) + 1 = Cast(next_start AS DATE) THEN 0 ELSE 1 END AS flag
   FROM
    ( -- fixing the bad data first
      SELECT t.*
        ,CASE WHEN RECORD_START_DT > RECORD_END_DT THEN RECORD_START_DT - INTERVAL '1' DAY ELSE RECORD_START_DT END AS fixed_start
      FROM tab AS t
    ) AS fixed_data
   QUALIFY flag = 1
 ) AS dt

This looks for gaps, after applying flag = 1 the current row got the max end date and the previous row the matching start date. The outer Select finally adds the this start date to the current row.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.