0

I am using Amazon Athena engine version 1, which is based on Presto 0.172.

Consider the example data set:

id date_column col1
1 01/03/2021 NULL
1 02/03/2021 1
1 15/03/2021 2
1 16/03/2021 NULL
1 17/03/2021 NULL
1 30/03/2021 NULL
1 30/03/2021 1
1 31/03/2021 NULL

I would like to replace all NULLs in the table with the last non-NULL value i.e. I want to get:

id date_column col1
1 01/03/2021 NULL
1 02/03/2021 1
1 15/03/2021 2
1 16/03/2021 2
1 17/03/2021 2
1 30/03/2021 1
1 30/03/2021 1
1 31/03/2021 1

I was thinking of using a lag function with IGNORE NULLS option but unfortunately, IGNORE NULLS is not supported by Athena engine version 1 (it is also not supported by Athena engine version 2, which is based on Presto 0.217).

How to achieve the desired format without using the IGNORE NULLS option?

Here is some template for generating the example table:

WITH source1 AS (
SELECT
 *
FROM (
  VALUES
  (1, date('2021-03-01'), NULL),
  (1, date('2021-03-02'), 1),
  (1, date('2021-03-15'), 2),
  (1, date('2021-03-16'), NULL),
  (1, date('2021-03-17'), NULL),
  (1, date('2021-03-30'), NULL), 
  (1, date('2021-03-30'), 1), 
  (1, date('2021-03-31'), NULL)
) AS t (id, date_col, col1)
)

SELECT
  id
, date_col
, col1
-- This doesn't work as IGNORE NULLS is not supported.
-- CASE 
--   WHEN col1 IS NOT NULL THEN col1
--   ELSE lag(col1) OVER IGNORE NULLS (PARTITION BY id ORDER BY date_col) 
-- END AS col1_lag_nulls_ignored
FROM
  source1
ORDER BY
  date_co

1 Answer 1

1

After reviewing similar questions on SO (here and here), the below solution will work for all column types (including Strings and dates):

WITH source1 AS (
SELECT
 *
FROM (
  VALUES
  (1, date('2021-03-01'), NULL),
  (1, date('2021-03-02'), 1),
  (1, date('2021-03-15'), 2),
  (1, date('2021-03-16'), NULL),
  (1, date('2021-03-17'), NULL),
  (1, date('2021-03-30'), 1), 
  (1, date('2021-03-31'), NULL)
) AS t (id, date_col, col1)
)

, grouped AS (
SELECT
  id
, date_col
, col1
-- If the row has a value in a column, then this row and all subsequent rows 
-- with a NULL (before the next non-NULL value) will be in the same group.
, sum(CASE WHEN col1 IS NULL THEN 0 ELSE 1 END) OVER (
    PARTITION BY id ORDER BY date_col) AS grp
FROM 
 source1
)

SELECT
  id
, date_col
, col1
-- max is used instead of first_value, since in cases where there will
-- be multiple records with NULL on the same date, the first_value may 
-- still return a NULL.
, max(col1) OVER (PARTITION BY id, grp ORDER BY date_col) AS col1_filled
, grp
FROM
  grouped 
ORDER BY
  date_col

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.