3
Total time: 1.01876 s
Function: prepare at line 91

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    91                                           @profile
    92                                           def prepare():
    93                                           
    94         1       5681.0   5681.0      0.6     
    95         1       2416.0   2416.0      0.2      
    96                                           
    97                                               
    98         1        536.0    536.0      0.1      tss = df.groupby('user_id').timestamp
    99         1     949643.0 949643.0     93.2      delta = tss.diff()
   100         1       1822.0   1822.0      0.2      
   101         1      13030.0  13030.0      1.3      
   102         1       5193.0   5193.0      0.5      
   103         1       1251.0   1251.0      0.1      
   104                                           
   105         1       2038.0   2038.0      0.2      
   106                                           
   107         1       1851.0   1851.0      0.2     
   108                                           
   109         1        282.0    282.0      0.0      
   110                                           
   111         1       3088.0   3088.0      0.3      
   112         1       2943.0   2943.0      0.3      
   113         1        438.0    438.0      0.0      
   114         1       4658.0   4658.0      0.5      
   115         1      17083.0  17083.0      1.7      
   116         1       3115.0   3115.0      0.3      
   117         1       3691.0   3691.0      0.4      
   118                                           
   119         1          2.0      2.0      0.0      

I have a dataframe which I group by some key and then select a column from each group and perform diff on that column (per group). As shown in the profiling results, the diff operation is super slow compared to the rest and is kind of a bottleneck. Is this expected? Are there faster alternatives to achieve the same result?

Edit: some more explanation In my use case timestamps represent the times for some actions of a user to which I want to calculate the deltas between these actions (they are sorted) but each user's actions are completely independent of other users.

Edit: Sample code

import pandas as pd
import numpy as np


df = pd.DataFrame(
    {'ts':[1,2,3,4,60,61,62,63,64,150,155,156,
           1,2,3,4,60,61,62,63,64,150,155,163,
           1,2,3,4,60,61,62,63,64,150,155,183],
    'id': [1,2,3,4,60,61,62,63,64,150,155,156,
           71,72,73,74,80,81,82,83,64,160,165,166,
           21,22,23,24,90,91,92,93,94,180,185,186],
    'other':['x','x','x','','x','x','','x','x','','x','',
             'y','y','y','','y','y','','y','y','','y','',
             'z','z','z','','z','z','','z','z','','z',''],
    'user':['x','x','x','x','x','x','x','x','z','x','x','y',
            'y','y','y','y','y','y','y','y','x','y','y','x',
            'z','z','z','z','z','z','z','z','y','z','z','z']
    })



df.set_index('id',inplace=True)
deltas=df.groupby('user').ts.transform(pd.Series.diff)
4
  • 1
    This is because a vectorised function is used on each group individually. One way you can speed this up is to use numpy and avoid performing an operation separately for each group, i.e. bypassing groupby. For that, I recommend you share some code with example data.
    – jpp
    Commented May 24, 2018 at 13:55
  • 1
    added sample code
    – mkmostafa
    Commented May 24, 2018 at 14:02
  • how about sorting based on user and then calculate the difference without grouping? You should just be aware that some values won't make sense and you have to either remove them or recalculate them separately. However in large data this won't make much difference
    – anishtain4
    Commented May 24, 2018 at 14:14
  • it would be nice if one can still spot the jumps to new users and start with a nan value somehow
    – mkmostafa
    Commented May 24, 2018 at 14:25

1 Answer 1

2

If you do not wish to sort your data or drop down to numpy, then a significant performance improvement may be possible by changing your user series to Categorical. Categorical data is stored effectively as integer pointers.

In the below example, I see an improvement from 86ms to 59ms. This may improve further for larger datasets and where more users are repeated.

df = pd.concat([df]*10000)

%timeit df.groupby('user').ts.transform(pd.Series.diff)  # 86.1 ms per loop

%timeit df['user'].astype('category')                    # 23.4 ms per loop
df['user'] = df['user'].astype('category')
%timeit df.groupby('user').ts.transform(pd.Series.diff)  # 35.7 ms per loop

If you are performing multiple operations, then the one-off cost of converting to categorical can be discounted.

1
  • 1
    Good tip, will try it out. I see also a suggestion to sort by user in one comment
    – mkmostafa
    Commented May 24, 2018 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.