5

I have a dataset with the following first three columns. Include Basket ID (unique identifier), Sale amount (dollars) and date of the transaction. I want to calculate the following column for each row of the dataset, and I would like to it in Python.

Previous Sale of the same basket (if any); Sale Count to date for current basket; Mean To Date for current basket (if available); Max To Date for current basket (if available)

Basket  Sale   Date       PrevSale SaleCount MeanToDate MaxToDate
88      $15 3/01/2012                1      
88      $30 11/02/2012      $15      2         $23        $30
88      $16 16/08/2012      $30      3         $20        $30
123     $90 18/06/2012               1      
477     $77 19/08/2012               1      
477     $57 11/12/2012      $77      2         $67        $77
566     $90 6/07/2012                1      

I'm pretty new with Python, and I really struggle to find anything to do it in a fancy way. I've sorted the data (as above) by BasketID and Date, so I can get the previous sale in bulk by shifting forward by one for each single basket. No clue how to get the MeanToDate and MaxToDate in an efficient way apart from looping... any ideas?

2
  • What format is your current 'dataset' in (the three first columns)? Is it a file or are you using some sort of data structure currently?
    – askewchan
    Mar 18, 2013 at 23:55
  • sorry, i've forgot to mention. it come from a text file, but is stored in a pandas' dataframe. Mar 18, 2013 at 23:58

2 Answers 2

4

This should do the trick:

from pandas import concat
from pandas.stats.moments import expanding_mean, expanding_count

def handler(grouped):
    se = grouped.set_index('Date')['Sale'].sort_index()
    # se is the (ordered) time series of sales restricted to a single basket
    # we can now create a dataframe by combining different metrics
    # pandas has a function for each of the ones you are interested in!
    return  concat(
        {
            'MeanToDate': expanding_mean(se), # cumulative mean
            'MaxToDate': se.cummax(),         # cumulative max
            'SaleCount': expanding_count(se), # cumulative count
            'Sale': se,                       # simple copy
            'PrevSale': se.shift(1)           # previous sale
        },
        axis=1
     )

# we then apply this handler to all the groups and pandas combines them
# back into a single dataframe indexed by (Basket, Date)
# we simply need to reset the index to get the shape you mention in your question
new_df = df.groupby('Basket').apply(handler).reset_index()

You can read more about grouping/aggregating here.

6
  • The cumulative function are just awesome - Thanks! Why the assumption of not having more than one transaction per basket per day? Mar 19, 2013 at 2:03
  • If the index isn't unique concat won't be able to align the columns properly back together since there will be multiple values for some indices, if that makes sense.
    – mtth
    Mar 19, 2013 at 2:06
  • make sense...and unfortunately there are multiple dates for the same basket. When you're suggesting to use resample, is it within the "handler" or before for the dataframe? I hope i've been asking something that make sense, as it not really clear what resample is actually doing (need to get back home first!) Mar 19, 2013 at 2:38
  • Having checked on an example, it seems to work also for duplicate indices. concat apparently aligns data with the same index in the order they appear. I edited my answer to account for that.
    – mtth
    Mar 19, 2013 at 3:01
  • you're the man, this is all great, powerful and superquick...it took 5 sec to process over 300k rows - thanks heaps! just last thing, why the reset_index at the bottom? i can't appreciate any difference between using it or no.. am i missing something? Mar 19, 2013 at 6:50
0


import pandas as pd
pd.__version__  # u'0.24.2'

from pandas import concat

def handler(grouped):
    se = grouped.set_index('Date')['Sale'].sort_index()
    return  concat(
        {
            'MeanToDate': se.expanding().mean(),   # cumulative mean
            'MaxToDate': se.expanding().max(),  # cumulative max
            'SaleCount': se.expanding().count(),   # cumulative count
            'Sale': se,                # simple copy
            'PrevSale': se.shift(1)   # previous sale
        },
        axis=1
     )

###########################
from datetime import datetime  
df = pd.DataFrame({'Basket':[88,88,88,123,477,477,566],
                  'Sale':[15,30,16,90,77,57,90],
                  'Date':[datetime.strptime(ds,'%d/%m/%Y') 
                          for ds in ['3/01/2012','11/02/2012','16/08/2012','18/06/2012',
                                    '19/08/2012','11/12/2012','6/07/2012']]})
#########

new_df = df.groupby('Basket').apply(handler).reset_index()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.