0

I have a dataframe like below:

product_id        date         status
    1          2018-09-11        G
    1          2016-01-11        B
    1          2018-02-11        P
    2          2019-06-12        P
    2          2020-10-11        P
    3          2019-07-21        G
    3          2016-09-11        B
    3          2018-12-15        B
    3          2020-11-07        P
    :
    :
    n          2020-11-11        G

Since I have duplicate product_id, I am thinking to group by the product id and only filter out the earliest date of the corresponding product_id, but my date column is not a datetime object, so I need to convert it to datetime first or I do not have to if there's other way to do it. The output dataframe should look like below:

product_id        date         status
    1          2016-01-11        B
    2          2019-06-12        P
    3          2016-09-11        B
    :
    :
    n          2020-11-11        G

I am wondering are there any better or faster ways to achieve my goal other than using group by or maybe drop duplicate?

2 Answers 2

2

Use to_datetime with DataFrameGroupBy.idxmin for index of minimal datetime, what is earliest and select values by DataFrame.loc:

df['date'] = pd.to_datetime(df['date'])

df = df.loc[df.groupby('product_id')['date'].idxmin()]
print (df)
   product_id       date status
1           1 2016-01-11      B
3           2 2019-06-12      P
6           3 2016-09-11      B

Detail: idxmin return indices by product_id with minimal date:

print (df.groupby('product_id')['date'].idxmin())
product_id
1    1
2    3
3    6
Name: date, dtype: int64

Or use DataFrame.sort_values with DataFrame.drop_duplicates:

df['date'] = pd.to_datetime(df['date'])

df = df.sort_values(['product_id','date']).drop_duplicates('product_id')
print (df)
   product_id       date status
1           1 2016-01-11      B
3           2 2019-06-12      P
6           3 2016-09-11      B
4
  • 1
    It works, thank you so much! And if you do not mind me asking, would you mind explain what is idxmin and how it works coz I am not very familiar with it Sep 18, 2020 at 5:21
  • @CodingStark idxmin & idxmax is pandas' take on argmax en.wikipedia.org/wiki/Arg_max
    – Paul H
    Sep 18, 2020 at 5:25
  • @CodingStark - I add some details to answer to see how working idxmin here by example data.
    – jezrael
    Sep 18, 2020 at 5:35
  • 1
    @jezrael Thank you so much! Glad that I am learning new things!!! Sep 18, 2020 at 5:43
1

Your dates are in YYYY-MM-DD order, the ISO-8601 definition. So you can simply sort on it:

print (df.sort_values("date").drop_duplicates("product_id").sort_index())

   product_id        date status
1           1  2016-01-11      B
3           2  2019-06-12      P
6           3  2016-09-11      B
1
  • 1
    I I think because performance always is best convert dates to datetimes.
    – jezrael
    Sep 18, 2020 at 5:23

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.