13

I'm merging overlapping intervals in pandas dataframe, and looking for efficient way to do it in pandas, besides the regular algorithms of running on rows 1 by 1. how can I do it in pandas?

I have tried regular overlapping algo of running on each row and ask if current row.start < last_end. and this worked for me.

Say I have input of the following dataframe

df:
    START   FINISH
0   0.000000    10.000000
1   10.000000   8700.182997
2   0.000000    10.000000
3   10.000000   9720.687227
4   9850.123    9990.000000

I would expect the output to be as following:

df:
    START   FINISH
0   0.000000    9720.687227
2   9850.123    9990.000000

Thanks in advance!

2 Answers 2

14

You can do it using only pandas

import pandas as pd
import io

## load data

raw ="""START,FINISH
0.000000    ,10.000000
10.000000   ,4500.182997
5000.00    ,7000.000000
6000   ,8500.687227
9850.123,9990.000000
"""

buf_bytes = io.StringIO(raw)
df=pd.read_csv(buf_bytes)

## solution

df.sort_values("START", inplace=True)

## This line compares if START of next row is greater than FINISH of current
## row ("shift" shifts down FINISH by one row). The value of expression before
## cumsum will be True if interval breaks (i.e. cannot be merged), so  
## cumsum will increment group value when interval breaks (cum sum treats True=1, False=0)
df["group"]=(df["START"]>df["FINISH"].shift()).cumsum()

## this returns min value of "START" column from a group and max value fro m "FINISH"
result=df.groupby("group").agg({"START":"min", "FINISH": "max"})
display(result)

output

 START       FINISH
group                       
0         0.000  4500.182997
1      5000.000  8500.687227
2      9850.123  9990.000000
8
  • can you please add some explanation on the solution ? for example, pandas function you used and their affect on results?
    – Saeed isa
    Commented Sep 11, 2019 at 7:13
  • you are welcome, I have added few comments on solution above hope that helps
    – Dev Khadka
    Commented Sep 11, 2019 at 7:24
  • it does! all clear, thanks, one last question, how can I get rid of "group" in results ?
    – Saeed isa
    Commented Sep 11, 2019 at 7:32
  • set index name blank result.index.name = ""
    – Dev Khadka
    Commented Sep 11, 2019 at 7:36
  • sorry I meant the whole column under group
    – Saeed isa
    Commented Sep 11, 2019 at 7:56
10

The above answer is inspiring but there is something need to be improved.

(1) It should be documented that shift() will shift up one record, not down. (2) It does not consider when a row is within the boundary of the previous record. Just add cummax() and will solve.

Here is the modified code:

import pandas as pd
import io

## load data

raw ="""START,FINISH
0.000000    ,10.000000
2.000000    ,3.000000
10.000000   ,4500.182997
5000.00    ,7000.000000
6000   ,8500.687227
9850.123,9990.000000
"""

buf_bytes = io.StringIO(raw)
df=pd.read_csv(buf_bytes)

## solution

df.sort_values("START", inplace=True)

## This line compares if START of present row is greater than largest FINISH in previous 
## rows ("shift" shifts up FINISH by one row). The value of expression before
## cumsum will be True if interval breaks (i.e. cannot be merged), so
## cumsum will increment group value when interval breaks (cum sum treats True=1, False=0)


df["group"]=(df["START"]>df["FINISH"].shift().cummax()).cumsum()

print(df)

## this returns min value of "START" column from a group and max value fro m "FINISH"
result=df.groupby("group").agg({"START":"min", "FINISH": "max"})
print(result)

output:

          START       FINISH
group                       
0         0.000  4500.182997
1      5000.000  8500.687227
2      9850.123  9990.000000

Results from the unmodified solution:

          START       FINISH
group                       
0         0.000    10.000000
1        10.000  4500.182997
2      5000.000  8500.687227
3      9850.123  9990.000000

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.