0

I am trying to merge two datasets: Events and mortality by patientid. Events have all the patientid including the patientid in mortality, but didn't have the timestamp and label (always equal 1) in mortality. I want to extract all records in events but not in mortality.

-- load events file
events = LOAD '../sample_test/sample_events.csv' USING PigStorage(',') AS (patientid:int, eventid:chararray, eventdesc:chararray, timestamp:chararray, value:float);

-- select required columns from events
events = FOREACH events GENERATE patientid, eventid, ToDate(timestamp, 'yyyy-MM-dd') AS etimestamp, value;

-- load mortality file
mortality = LOAD '../sample_test/sample_mortality.csv' USING PigStorage(',') as (patientid:int, timestamp:chararray, label:int);

mortality = FOREACH mortality GENERATE patientid, ToDate(timestamp, 'yyyy-MM-dd') AS mtimestamp, label;

--To display the relation, use the dump command e.g. DUMP mortality;

-- ***************************************************************************
-- Compute the index dates for dead and alive patients
-- ***************************************************************************
eventswithmort = JOIN events BY patientid, mortality BY patientid;-- perform join of events and mortality by patientid;
eventswithmort = FOREACH eventswithmort GENERATE events.patientid, events.eventid, events.etimestamp, events.value, mortality.mtimestamp, mortality.label;

aliveevents = FILTER eventswithmort BY mortality::label != 1;

My question is for last row of code should I use label != 1 or label is null? It seems that I always got an empty dataset.

1 Answer 1

1
eventswithmort = FOREACH eventswithmort GENERATE events.patientid, events.eventid, events.etimestamp, events.value, mortality.mtimestamp, mortality.label;

not totally sure this is the syntax for generating the data, instead of . should you not use ::

if you want rows in events but NOT in mortality then the join should be left outer, pig default is an inner join.

eventswithmort = JOIN events BY patientid LEFT OUTER, mortality BY patientid;

then for each eventswithmort you can FILTER by mortality::label IS NULL;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.