2

So I have 2 dataframes that I'm joining by their redefined index which is the number we use to identify the study, when I'm joining them they look like this:

df1 (contains all study numbers):

Index State PS
1001 CA 0
1002 NY 0
1003 NJ 1

df2 (does not contain all study numbers and contains duplicates):

Index Study
1001 Active
1002 Active
1002 Closed

I currently have df1 = df1.join(df2) which outputs:

Index State PS Study
1001 CA 0 Active
1002 NY 0 Active
1002 NY 0 Closed
1003 NJ 1

In this example df, I'd like only the first instance of 1002 in df2 to be merged with df1. Assuming it has something to do with 'how' or 'on', but I don't understand the documentation well enough as I am pretty new to Pandas. Thanks! Desired output is:

Index State PS Study
1001 CA 0 Active
1002 NY 0 Active
1003 NJ 1
5
  • Is it a coincidence that the first record 1002 is active in the column study or is it true for all entries you define as first instance ? Sep 24, 2021 at 1:18
  • Why Active over Closed for 1002? What is the logic here? Sep 24, 2021 at 1:22
  • Not all first instances will have 'Active' (i.e. 1002 1st instance could be Closed, 2nd instance Active, 3rd Instance Active) but I only want the first instance as that is the most recent one. Basically DF2 comes from a data set that is sorted newest to oldest, but with no date. Sep 24, 2021 at 1:24
  • @ScottBoston see my above comment. Basically DF2 comes from a 2 column data set that only contains the study # and study status (active or closed). The first instance is the most recent status. It's dumb that there is no date, or the prior instance is not deleted when the status is updated, but I have no control over that. It is only a general log Sep 24, 2021 at 1:26
  • @slicedorange7 do approve the solution below if it helps you :) Sep 24, 2021 at 1:37

2 Answers 2

1

Try using drop_duplicates with keep="first" since it is sorted from newest to oldest. Then you merge on the key Index

df2 = df2.drop_duplicates(subset="Index", keep="first")
df = pd.merge(df1, df2, on="Index", how="left")
1
  • Thanks, pretty much used this same thing except for df2 I used: df2 = df2[~df2.index.duplicated(keep = "first")] Sep 24, 2021 at 1:50
-1
import pandas as pd

dict1 = {
    'State': ['CA', 'NY', 'NJ'],
    'PS': [0, 0, 1]
}
dict2 = {
    'Study': ['Active', 'Active', 'Closed'],
}

df1 = pd.DataFrame(data=dict1, index=[1001, 1002, 1003])
df2 = pd.DataFrame(data=dict2, index=[1001, 1002, 1002])

print(df1)
print(df2)

answer = df1.join(df2).drop_duplicates(subset=['State', 'PS'], keep='first')

print(answer)

Crucially, the drop_duplicates method should be able to handle this special case

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.