18

How can I calculate the time (number of days) between "events" in a Pandas time series? For example, if I have the below time series I'd like to know on each day in the series how many days have passed since the last TRUE

            event
2010-01-01  False
2010-01-02   True
2010-01-03  False
2010-01-04  False
2010-01-05   True
2010-01-06  False

The way I've done it seems overcomplicated, so I'm hoping for something more elegant. Obviously a for loop iterating over the rows would work, but I'm looking for a vectorized (scalable) solution ideally. My current attempt below:

date_range = pd.date_range('2010-01-01', '2010-01-06')
df = pd.DataFrame([False, True, False, False, True, False], index=date_range, columns=['event'])
event_dates = df.index[df['event']]
df2 = pd.DataFrame(event_dates, index=event_dates, columns=['max_event_date'])
df = df.join(df2)
df['max_event_date'] = df['max_event_date'].cummax(axis=0, skipna=False)
df['days_since_event'] = df.index - df['max_event_date']

            event max_event_date  days_since_event
2010-01-01  False            NaT               NaT
2010-01-02   True     2010-01-02            0 days
2010-01-03  False     2010-01-02            1 days
2010-01-04  False     2010-01-02            2 days
2010-01-05   True     2010-01-05            0 days
2010-01-06  False     2010-01-05            1 days

5 Answers 5

8

Continuing to improve on this answer, and hoping that someone comes in with 'the' pythonic way. Until then, I think this final update works best.

last = pd.to_datetime(np.nan)
def elapsed(row):
    if not row.event:
        return row.name - last
    else:
        global last
        last = row.name
        return row.name-last

df['elapsed'] = df.apply(elapsed,axis=1)

df
            event  elapsed
2010-01-01  False      NaT
2010-01-02   True   0 days
2010-01-03  False   1 days
2010-01-04  False   2 days
2010-01-05   True   0 days
2010-01-06  False   1 days

:::::::::::::

Leaving previous answers below although they are sub-optimal

:::::::::

Instead of making multiple passes through, seems easier to to just loop through the indexes

df['elapsed'] = 0
for i in df.index[1:]:
    if not df['event'][i]:
        df['elapsed'][i] = df['elapsed'][i-1] + 1

::::::::::::

Let's say 'Trues' are your event of interest.

trues = df[df.event==True]
trues.Dates = trues.index #need this because .diff() doesn't work on the index
trues.Elapsed = trues.Dates.diff()
3

I run into groupby().diff() lately which can offer the following method:

  1. Use groupby.diff to calculate the days to last True day:

    df.loc[df.index[-1]+pd.Timedelta(days=1), 'event'] = True  # add an artificial True day for interpolation
    df['last']=df.index
    df['last']=df.groupby('event')['last'].diff()
    df.loc[df['event']==False, 'last'] = None
    

    which gives you:

                event   last
    2010-01-01  False   NaT
    2010-01-02  True    NaT
    2010-01-03  False   NaT
    2010-01-04  False   NaT
    2010-01-05  True    3 days
    2010-01-06  False   NaT
    2010-01-07  True    2 days
    
  2. Use tshift() to set correct last value for True and the False before:

    df['last'] = (df['last']-pd.Timedelta(days=1)).tshift(periods=-1, freq='D')
    df.loc[df['event'], ['last']] = pd.Timedelta(days=0)
    

    you will get:

                event   last
    2010-01-01  False   NaT
    2010-01-02  True    0 days
    2010-01-03  False   NaT
    2010-01-04  False   2 days
    2010-01-05  True    0 days
    2010-01-06  False   1 days
    2010-01-07  True    0 days
    
  3. Lastly interpolate the NaN values linearly to get the final result

    df['last'] /= np.timedelta64(1, 'D')
    df.interpolate(method='linear', axis=0, inplace=True)
    df.drop(df.index[-1], inplace=True)  # erase the artificial row
    df['last'] *= np.timedelta64(1, 'D')
    
                event   last
    2010-01-01  False   NaN
    2010-01-02  True    0 days
    2010-01-03  False   1 days
    2010-01-04  False   2 days
    2010-01-05  True    0 days
    2010-01-06  False   1 days
    
3

A one-pass solution would certainly be ideal, but here's a multi-pass solution using only (presumably) cythonized pandas functions:

def get_delay(ds):
    x1 = (~ds).cumsum()
    x2 = x1.where(ds, np.nan).ffill()
    return x1 - x2

date_range = pd.date_range('2010-01-01', '2010-01-06')
ds = pd.Series([False, True, False, False, True, False], index=date_range)
pd.concat([ds, get_delay(ds)], axis=1)


            Event   Last
2010-01-01  False   NaN
2010-01-02  True    0
2010-01-03  False   1
2010-01-04  False   2
2010-01-05  True    0
2010-01-06  False   1

And interestingly it seems to perform a little better in some quick benchmarks, possibly due to avoiding row-wise operations:

%%timeit -n 1000

    def get_delay(ds):
        x1 = (~ds).cumsum()
        x2 = x1.where(ds, np.nan).ffill()
        return x1 - x2

    n = 100
    events = np.random.choice([True, False], size=n)
    date_range = pd.date_range('2010-01-01', periods=n)
    df = pd.DataFrame(events, index=date_range, columns=['event'])
    get_delay(df['event'])

1000 loops, best of 3: 1.09 ms per loop

Versus the single loop approach with a global:

%%timeit -n 1000

last = pd.to_datetime(np.nan)
def elapsed(row):
    if not row.event:
        return row.name - last
    else:
        global last
        last = row.name
        return row.name-last


n = 100
events = np.random.choice([True, False], size=n)
date_range = pd.date_range('2010-01-01', periods=n)
df = pd.DataFrame(events, index=date_range, columns=['event'])
df.apply(elapsed, axis=1)

1000 loops, best of 3: 2.4 ms per loop

Perhaps there's some nuance in that comparison that doesn't make it fair but either way, the no-custom-functions version certainly doesn't seem to be a whole lot slower, if at all.

1

Here another approach, comparing the dates with a lookup table.


import pandas as pd
import io

data=io.StringIO('''
date,event
2010-01-01,False
2010-01-02,True
2010-01-03,False
2010-01-04,False
2010-01-05,True
2010-01-06,False
''')

df = pd.read_csv( data, parse_dates=['date'] )
df.set_index( 'date', inplace=True )
print( df )

            event
date             
2010-01-01  False
2010-01-02   True
2010-01-03  False
2010-01-04  False
2010-01-05   True
2010-01-06  False

I first make a list of the dates when the event happened:

when_events = df[ (df['event']==True) ].index
when_events = pd.Series( when_events )
print( when_events )

0   2010-01-02
1   2010-01-05
Name: date, dtype: datetime64[ns]

And then use it to lookup the biggest date that is not bigger than my index:

df[ 'last' ] = df.index
df[ 'last' ] = df['last'].apply( lambda x: when_events[ when_events<=x ].max() )
df[ 'elapsed' ] = df.index.values - df[ 'last' ] 
print( df )

            event       last elapsed
date                                
2010-01-01  False        NaT     NaT
2010-01-02   True 2010-01-02  0 days
2010-01-03  False 2010-01-02  1 days
2010-01-04  False 2010-01-02  2 days
2010-01-05   True 2010-01-05  0 days
2010-01-06  False 2010-01-05  1 days

I am sure it can be prettier and smaller, but you get the idea.

Hope it helps!


0

If someone is looking for a readable, simple solution that is perhaps not efficient on larger datasets, I just did the following. In my setting, I wanted to calculate the number of speaker utterances (turns) between topic changes in a conversation. coder referred to a particular research assistant (many research assistants coded each conversation, so each had their own column of 1's and 0's indicating topic changes or topic continuations). In my case, adjacent rows always differed by one time step, so I didn't need to access a datetime index -- I could just increment (and reset at topic changes) a turns_since_last counter on every new row (utterance/turn) in my dataset:

def turns_since_last_topic(coder):
    turns_since_last = 0
    coding['turns_since_last_{}'.format(coder)] = np.nan    
    for idx, row in coding.iterrows():
        if not row[coder]:
            turns_since_last += 1
        else:
            turns_since_last += 1
            coding.loc[idx, 'turns_since_last_{}'.format(coder)] = turns_since_last
            turns_since_last = 0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.