I am trying to merge two datasets: Events
and mortality
by patientid
. Events have all the patientid
including the patientid in mortality
, but didn't have the timestamp
and label
(always equal 1) in mortality
. I want to extract all records in events
but not in mortality
.
-- load events file
events = LOAD '../sample_test/sample_events.csv' USING PigStorage(',') AS (patientid:int, eventid:chararray, eventdesc:chararray, timestamp:chararray, value:float);
-- select required columns from events
events = FOREACH events GENERATE patientid, eventid, ToDate(timestamp, 'yyyy-MM-dd') AS etimestamp, value;
-- load mortality file
mortality = LOAD '../sample_test/sample_mortality.csv' USING PigStorage(',') as (patientid:int, timestamp:chararray, label:int);
mortality = FOREACH mortality GENERATE patientid, ToDate(timestamp, 'yyyy-MM-dd') AS mtimestamp, label;
--To display the relation, use the dump command e.g. DUMP mortality;
-- ***************************************************************************
-- Compute the index dates for dead and alive patients
-- ***************************************************************************
eventswithmort = JOIN events BY patientid, mortality BY patientid;-- perform join of events and mortality by patientid;
eventswithmort = FOREACH eventswithmort GENERATE events.patientid, events.eventid, events.etimestamp, events.value, mortality.mtimestamp, mortality.label;
aliveevents = FILTER eventswithmort BY mortality::label != 1;
My question is for last row of code should I use label != 1
or label is null
? It seems that I always got an empty dataset.