1

I have a sales table with custid, transaction date column, etc. I am using groupby on the custid column, and then using the agg method to get the max date( to get the latest date of transaction by that particular customer) and min date ( to get the first date of his transaction at the shop).

My code is as below:

sales['transdate'] = pd.to_datetime(sales['transdate']) # Converting the transdate column from string to timestamps.
sales['custid'].groupby.transdate({'count': np.count_nonzero ,'first': np.min, 'last' : np.max})

I would like to know whether its okay to

calculate min and max between dates by using np.min/max methods. or should I be using some other datetime related methods?

0

1 Answer 1

1

You should use groupby.agg to apply multiple aggregation functions.

Note also that with Pandas many aggregation functions can be called via strings. In this case, you can use 'size', 'min' and 'max'. Using strings is recommended since the string representation is mapped by Pandas to tested and efficient algorithms.

Here's a demo:

df = pd.DataFrame([['2017-01-14', 1], ['2017-12-05', 2], ['2017-06-15', 2],
                   ['2017-03-21', 1], ['2017-04-25', 2], ['2017-02-12', 1]],
                  columns=['transdate', 'custid'])

df['transdate'] = pd.to_datetime(df['transdate'])

agg_dict = {'count': 'size', 'first': 'min', 'last': 'max'}

res = df.groupby('custid')['transdate'].agg(agg_dict)

print(res)

        count      first       last
custid                             
1           3 2017-01-14 2017-03-21
2           3 2017-04-25 2017-12-05
4
  • I thought that using numpy methods were more efficient. And, what will be the trade-off between using numpy or string methods.
    – aspiring1
    Jul 4, 2018 at 16:19
  • @aspiring1, Usually, the strings point to NumPy methods. The reason I state a preference for strings is you know they've been tested. For example, we trust Pandas methods with df.groupby('a')['b'].mean() (I've never seen anyone specify np.mean explicitly for this), so there's no reason to mistrust Pandas with "mean" as an argument. A non-NumPy performance example is len vs "size", the string is usually faster.
    – jpp
    Jul 4, 2018 at 16:21
  • I am always confused as to whether df.groupby('a')['b'].mean() and using numpy.mean of the above have the same speed.
    – aspiring1
    Jul 4, 2018 at 16:27
  • @aspiring1, Complexity should be identical. There may be a fixed O(1) cost attached to using the Pandas method, but this should be negligible. If this kind of optimization really matters, you should consider working in pure NumPy. Most Pandas operations can be optimized if you extract NumPy arrays from your dataframe.
    – jpp
    Jul 4, 2018 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.