I have a dataframe that contains data like below (tiny subset of data):
I'm trying to figure out a way where I can create a new dataframe that contains all rows that have the same values for : carrier
, flightnumber
, departureAirport
and arrivalAirport
but also have date ranges that overlap.
By overlap I mean the effectiveDate
for one row falls between the effectiveDate
and discontinuedDate
for another record that has the same values for the other columns I mentioned.
So in my above example, the first two rows would be considered an example of this (and should both be included in the new dataframe), but the third row is not.
I'm assuming I want to use groupby, but I'm not entirely clear on what aggregation function I would apply. Below is what I have so far:
df.groupby(['carrier','flightnumber','departureAirport','arrivalAirport'])['effectiveDate', 'discontinuedDate'].min()
but obviously I need to apply a function that determines overlap instead of min()
. How would I go about identifying overlap instead of returning the minimum values for this group?
UPDATE:
carrier flightnumber departureAirport arrivalAirport effectiveDate discontinuedDate
4U 9748 DUS GVA 2017-05-09 2017-07-12
4U 9748 DUS GVA 2017-05-14 2017-07-16
4U 9748 DUS GVA 2017-07-18 2017-08-27
AG 1234 SFO DFW 2017-03-09 2017-05-12
AG 1234 SFO DFW 2017-03-14 2017-05-16
UPDATE 2:
As far as output goes I'd like to have any rows that overlap and have the same values for carrier
, flightnumber
, departureAirport
and arrivalAirport
returned in a new dataframe. There does not need to be any additional data included for these rows. So for the above example data, a dataframe like the one below would be my desired output:
carrier flightnumber departureAirport arrivalAirport effectiveDate discontinuedDate
4U 9748 DUS GVA 2017-05-09 2017-07-12
4U 9748 DUS GVA 2017-05-14 2017-07-16
AG 1234 SFO DFW 2017-03-09 2017-05-12
AG 1234 SFO DFW 2017-03-14 2017-05-16
Notice that only one record has been excluded (the third for 9748
) - this is because it's date range does not overlap with other records for the same flight.