126

How to group values of pandas dataframe and select the latest(by date) from each group?

For example, given a dataframe sorted by date:

    id     product   date
0   220    6647     2014-09-01 
1   220    6647     2014-09-03 
2   220    6647     2014-10-16
3   826    3380     2014-11-11
4   826    3380     2014-12-09
5   826    3380     2015-05-19
6   901    4555     2014-09-01
7   901    4555     2014-10-05
8   901    4555     2014-11-01

grouping by id or product, and selecting the earliest gives:

    id     product   date
2   220    6647     2014-10-16
5   826    3380     2015-05-19
8   901    4555     2014-11-01

6 Answers 6

155

You can also use tail with groupby to get the last n values of the group:

df.sort_values('date').groupby('id').tail(1)

    id  product date
2   220 6647    2014-10-16
8   901 4555    2014-11-01
5   826 3380    2015-05-19
5
  • 9
    I like this because it can be applied to more than just dates. Feb 14, 2018 at 16:28
  • 1
    This option is significantly faster, than the accepted answer, but is less readable. Also isn't it a problematic, that there is an assumption that groupby preserves order?
    – Michael D
    Jan 1, 2020 at 14:54
  • 4
    groupby preserves order, see stackoverflow.com/questions/26456125/… Apr 12, 2020 at 2:13
  • @ade1e how would the code change to perform a resample (say per month or year) and keep the last n values of the group, rather than summing/averaging?
    – Andreuccio
    Oct 7, 2020 at 13:14
  • 2
    I find this answer much more readable than the accepted one @MichaelD :) Dec 21, 2021 at 12:53
73

use idxmax in groupby and slice df with loc

df.loc[df.groupby('id').date.idxmax()]

    id  product       date
2  220     6647 2014-10-16
5  826     3380 2015-05-19
8  901     4555 2014-11-01
4
  • 8
    the solution works very slow for millions of records Jan 3, 2020 at 10:30
  • In 2021 I get this error: KeyError: "Passing list-likes to .loc or [] with any missing labels is no longer supported.
    – kame
    Apr 8, 2021 at 6:21
  • Use reindex instead
    – piRSquared
    Apr 8, 2021 at 6:24
  • will this not work if date column has NaTs? Jul 2, 2021 at 8:03
35

I had a similar problem and ended up using drop_duplicates rather than groupby.

It seems to run significatively faster on large datasets when compared with other methods suggested above.

df.sort_values(by="date").drop_duplicates(subset=["id"], keep="last")

    id  product        date
2  220     6647  2014-10-16
8  901     4555  2014-11-01
5  826     3380  2015-05-19
2
  • 4
    I typically use this as well, but wish the faster solution was with the groupby. The groupby intuitively makes more sense and is usually how we think about solving this problem!
    – rmilletich
    Dec 10, 2019 at 6:44
  • This approach, however, only works if you want to keep 1 record per group, rather than N records when using tail as per @nipy's answer
    – npetrov937
    Dec 5, 2022 at 9:42
25

Given a dataframe sorted by date, you can obtain what you ask for in a number of ways:

Like this:

df.groupby(['id','product']).last()

like this:

df.groupby(['id','product']).nth(-1)

or like this:

df.groupby(['id','product']).max()

If you don't want id and product to appear as index use groupby(['id', 'product'], as_index=False). Alternatively use:

df.groupby(['id','product']).tail(1)
1
  • 6
    In my tests, last() behaves a bit differently than nth(), when there are None values in the same column. For example, if first row in a group has the value 1 and the rest of the rows in the same group all have None, last() will return 1 as the value, although the last row has None. On the other hand nth(-1) will return None, which is more like what I expect. Oct 5, 2021 at 15:25
4

To use .tail() as an aggregation method and keep your grouping intact:

df.sort_values('date').groupby('id').apply(lambda x: x.tail(1))

        id  product date
id              
220 2   220 6647    2014-10-16
826 5   826 3380    2015-05-19
901 8   901 4555    2014-11-01
0
#import datetime library
from datetime import datetime as dt

#transform the date column to ordinal, or create a temp column converting to ordinal.
df['date'] = df.date.apply(lambda date: date.toordinal())

#apply aggregation function depending your desire. Earliest or Latest date.
latest_date = df.groupby('id').agg(latest=('date', max)) 
earliest_date = df.groupby('id').agg(earliest=('date', min)) 

#convert it from ordinal back to date.
df['date'] = df.date.apply(lambda date: dt.fromordinal(date))


#This operation may take seconds on millions of records.
1
  • Well, you certainly don't need to perform this conversion just to find the latest or earliest date.
    – AlexK
    Sep 30, 2022 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.