2

I'm finding some unexpected results. What I am trying to do is create a column that looks at the ID number and the date, and will count how many times that ID number comes up in the last 7 days (I'd also like to make that dynamic for an x amount of days, but just trying out with 7 days).

So given this dataframe:

import pandas as pd



df = pd.DataFrame(
        [['A', '2020-02-02 20:31:00'],
        ['A', '2020-02-03 00:52:00'],
        ['A', '2020-02-07 23:45:00'],
        ['A', '2020-02-08 13:19:00'],
        ['A', '2020-02-18 13:16:00'],
        ['A', '2020-02-27 12:16:00'],
        ['A', '2020-02-28 12:16:00'],
        ['B', '2020-02-07 18:57:00'],
        ['B', '2020-02-07 21:50:00'],
        ['B', '2020-02-12 19:03:00'],
        ['C', '2020-02-01 13:50:00'],
        ['C', '2020-02-11 15:50:00'],
        ['C', '2020-02-21 10:50:00']],
        columns = ['ID', 'Date'])

Code to calculate occurrence in last 7 days for each instance:

df['Date'] = pd.to_datetime(df['Date'])

delta = 7
df['count_in_last_%s_days' %(delta)] = df.groupby(['ID', pd.Grouper(freq='%sD' %delta, key='Date')]).cumcount()

Output:

   ID                Date  count_in_last_7_days
0   A 2020-02-02 20:31:00                     0
1   A 2020-02-03 00:52:00                     1
2   A 2020-02-07 23:45:00                     2
3   A 2020-02-08 13:19:00                     0 #<---- This should output 3
4   A 2020-02-18 13:16:00                     0
5   A 2020-02-27 12:16:00                     0
6   A 2020-02-28 12:16:00                     1
7   B 2020-02-07 18:57:00                     0
8   B 2020-02-07 21:50:00                     1
9   B 2020-02-12 19:03:00                     0 #<---- THIS SHOULD OUTPUT 2
10  C 2020-02-01 13:50:00                     0
11  C 2020-02-11 15:50:00                     0
12  C 2020-02-21 10:50:00                     0
2
  • Your example only spans one week (after we groupby ID), so we can't see that the 7-day window is working right. Can you make your example larger, to test that?
    – smci
    Mar 10, 2020 at 12:44
  • @smci, good point. updated above
    – chitown88
    Mar 10, 2020 at 12:56

2 Answers 2

4

You do not want to use a Grouper on Date but a rolling window. A grouper will segment the dataframe in separate consecutive blocks of the required duration. As you want 7 days from each date, this is the job of rolling:

delta = 7
df['count_in_last_%s_days' %(delta)] = df.assign(count=1).groupby(
    ['ID']).apply(lambda x: x.rolling('%sD' %delta, on='Date').sum(
        ))['count'].astype(int) - 1

it gives as expected:

   ID                Date  count_in_last_7_days
0   A 2020-02-02 20:31:00                     0
1   A 2020-02-03 00:52:00                     1
2   A 2020-02-07 23:45:00                     2
3   A 2020-02-08 13:19:00                     3
4   A 2020-02-18 13:16:00                     0
5   A 2020-02-27 12:16:00                     0
6   A 2020-02-28 12:16:00                     1
7   B 2020-02-07 18:57:00                     0
8   B 2020-02-07 21:50:00                     1
9   B 2020-02-12 19:03:00                     2
10  C 2020-02-01 13:50:00                     0
11  C 2020-02-11 15:50:00                     0
12  C 2020-02-21 10:50:00                     0
2
  • ok that makes sense why it would have that output. Thank you for not only the code, but the explanation.
    – chitown88
    Mar 10, 2020 at 13:53
  • Great answer. Tagged: rolling-computation. If the pandas doc doesn't adequately cover when you use rolling vs Grouper, a docbug is worth filing...
    – smci
    Mar 11, 2020 at 0:57
0

Looks like a rolling on Date with correct window will do:

(df.set_index('Date')
   .assign(count_last=1)
   .groupby('ID')
   .rolling(f'{delta}D')
   .sum() - 1
)

Output:

                        count_last
ID Date                           
A  2020-02-02 20:31:00         0.0
   2020-02-03 00:52:00         1.0
   2020-02-07 23:45:00         2.0
   2020-02-08 13:19:00         3.0
   2020-02-18 13:16:00         0.0
   2020-02-27 12:16:00         0.0
   2020-02-28 12:16:00         1.0
B  2020-02-07 18:57:00         0.0
   2020-02-07 21:50:00         1.0
   2020-02-12 19:03:00         2.0
C  2020-02-01 13:50:00         0.0
   2020-02-11 15:50:00         0.0
   2020-02-21 10:50:00         0.0
7
  • I tried to replicate the solution and it does not work. There is an error ValueError: window must be an integer 0 or greater
    – Duy Bui
    Apr 20, 2022 at 16:39
  • 1
    @DuyBui Date columns must be of Datetime type. Apr 20, 2022 at 17:48
  • You can try the example (with code) as in the first post with your code. It won't work. And yes, the Date is DateTime df['Date'] = pd.to_datetime(df['Date'])
    – Duy Bui
    Apr 20, 2022 at 18:50
  • 1
    Worked fine back then (accepted answer), works fine just now on my system. Make sure you have correct Pandas version. Check it out here as well. Apr 20, 2022 at 18:56
  • Sorry, I understand why it returned an error. I assigned it back to the original DataFrame while you changed the index of the newly created one.
    – Duy Bui
    Apr 20, 2022 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.