Vertica's shiny feature:
I have two tables that I would like to join with INTERPOLATE and expect data from the second table to be interpolated with the latest available. But unfortunately I am not able to get my desired result. I have checked out the Vertica documentation related to the INTERPOLATE feature and tried an example which worked fine.
CREATE TABLE a
( family int,
date DATE,
id int
);
CREATE TABLE b
( Id int,
date DATE,
datapoint float
);
INSERT INTO a VALUES (1, '20130603', 1);
INSERT INTO a VALUES (1, '20130604', 1);
INSERT INTO a VALUES (1, '20130605', 1);
INSERT INTO a VALUES (1, '20130606', 1);
INSERT INTO a VALUES (1, '20130607', 1);
INSERT INTO b VALUES (1, '20130603', 3.00);
SELECT a.family, a.date, a.id, a.date, b.datapoint
FROM a
LEFT
JOIN b
ON a.id = b.id
AND a.date INTERPOLATE PREVIOUS VALUE b.date;
vertdeva01:20130612-095628 > \g
family | date | id | date | data
--------+------------+----+------------+------
1 | 2013-06-03 | 1 | 2013-06-03 | 3
1 | 2013-06-04 | 1 | 2013-06-04 | 3
1 | 2013-06-05 | 1 | 2013-06-05 | 3
1 | 2013-06-06 | 1 | 2013-06-06 | 3
1 | 2013-06-07 | 1 | 2013-06-07 | 3
There I get the results as expected. The values in table b are interpolated according to the dates in table a.
But when I try something similar to a slightly more complex scenario I don't really get what I want.
What I intend to achieve:
What I intend to achieve is select latest available data from b for every id in a that matches the corresponding date in a. So if a has an (id,date) combination then I would like to fetch data from b for that id and date. But if there is no data for that id in b on that date, then fetch what is available AS OF that date. Fetch data that would be valid as of the date in a. In other words, a feel-back behavior. A data point for an id is valid in b as long as there is no other data point for that id after that date. I hope that makes sense. I know a way to do this using MAX()
and GROUP BY
. I would like to know if the same is possible using INTERPOLATE
An Example:
Just to give you an idea of what it is, I played with an example. This time I just modified the previously created tables to have more fields.
CREATE TABLE a
( family int,
family_name varchar(50),
industry varchar(15),
style_flag varchar(1),
id int,
id_name varchar(50),
id1 int,
id2 int,
id3 int,
date DATE,
id4 int
);
CREATE TABLE b
( id4 int,
flag int,
period int,
date DATE,
datapoint float
);
INSERT INTO a VALUES (1, '1family', 'comp', 'A', 1, '1 id', 101, 201, 301, '20130603', 401);
INSERT INTO a VALUES (1, '1family', 'comp', 'A', 2, '2 id', 102, 202, 302, '20130603', 402);
INSERT INTO a VALUES (1, '1family', 'comp', 'A', 3, '3 id', 103, 203, 303, '20130603', 403);
INSERT INTO a VALUES (2, '2family', 'bio', 'A', 5, '5 id', 105, 205, 305, '20130603', 405);
INSERT INTO a VALUES (2, '2family', 'bio', 'A', 7, '7 id', 107, 207, 307, '20130603', 407);
INSERT INTO a VALUES (2, '2family', 'bio', 'A', 9, '9 id', 109, 209, 309, '20130603', 409);
INSERT INTO b VALUES (401, 1, 10, '20130501', 2.00);
INSERT INTO b VALUES (401, 1, 20, '20130501', 1.50);
INSERT INTO b VALUES (401, 2, 10, '20130409', 12.34);
INSERT INTO b VALUES (401, 2, 20, '20130401', 10.56);
INSERT INTO b VALUES (402, 1, 10, '20130501', 2.00);
INSERT INTO b VALUES (402, 2, 20, '20130409', 12.34);
INSERT INTO b VALUES (402, 2, 20, '20130401', 10.56);
INSERT INTO b VALUES (402, 2, 20, '20130515', 20.55);
when I run the following query:
SELECT a.family, a.family_name, a.industry, a.style_flag,
a.id, a.id_name,
a.id1, a.id2, a.id3, a.date,
b.id4, b.flag, b.period, b.datapoint
FROM a
LEFT
JOIN b
ON a.id4 = b.id4
AND a.date INTERPOLATE PREVIOUS VALUE b.date;
I get the following:
family | family_name | industry | style_flag | id | id_name | id1 | id2 | id3 | date | id4 | flag | period | datapoint
--------+-------------+----------+------------+----+---------+-----+-----+-----+------------+-----+------+--------+-----------
2 | 2family | bio | A | 5 | 5 id | 105 | 205 | 305 | 2013-06-03 | | | |
1 | 1family | comp | A | 1 | 1 id | 101 | 201 | 301 | 2013-06-03 | 401 | 1 | 10 | 2
1 | 1family | comp | A | 3 | 3 id | 103 | 203 | 303 | 2013-06-03 | | | |
2 | 2family | bio | A | 9 | 9 id | 109 | 209 | 309 | 2013-06-03 | | | |
2 | 2family | bio | A | 7 | 7 id | 107 | 207 | 307 | 2013-06-03 | | | |
1 | 1family | comp | A | 2 | 2 id | 102 | 202 | 302 | 2013-06-03 | 402 | 2 | 20 | 20.55
But I need to select the latest value available for an id from b for a kind of group by of (id4,flag, period), instead of what it is currently giving me as a result. is there a way that I can make use of the INTERPOLATE feature for this? Or should I take a completely different approach. The problem is data in table b is sparse. We may not have a datapoint every day, where as in a we have a data point every day.
I also tried filling in the gaps between the data points in b using TIMESERIES clause and TS_FIRST_VALUE(datapoint, 'const'). But there again, the latest date available for a combination of id4, flag, period in b could be way back in time when compared to the date for an id in a. And I end up with the same problem as demonstrated above.
Any guidance would be highly appreciated.