20

How can I merge two pandas DataFrames on two columns with different names and keep one of the columns?

df1 = pd.DataFrame({'UserName': [1,2,3], 'Col1':['a','b','c']})
df2 = pd.DataFrame({'UserID': [1,2,3], 'Col2':['d','e','f']})
pd.merge(df1, df2, left_on='UserName', right_on='UserID')

This provides a DataFrame like this

enter image description here

But clearly I am merging on UserName and UserID so they are the same. I want it to look like this. Is there any clean ways to do this?

enter image description here

Only the ways I can think of are either re-naming the columns to be the same before merge, or droping one of them after merge. I would be nice if pandas automatically drops one of them or I could do something like

pd.merge(df1, df2, left_on='UserName', right_on='UserID', keep_column='left')
0

2 Answers 2

17

How about set the UserID as index and then join on index for the second data frame?

pd.merge(df1, df2.set_index('UserID'), left_on='UserName', right_index=True)

#   Col1    UserName    Col2
# 0    a           1       d
# 1    b           2       e
# 2    c           3       f
1
  • Great answer. I did almost exactly what OP did and got the redundant column. Reading from your code, I take it that if I join the left and right by index, that column will sort of 'merge' into the index column and therefore doesn't show up in the result? Thanks.
    – Bowen Liu
    Sep 25, 2018 at 18:26
10

There is nothing really nice in it: it's meant to be keeping the columns as the larger cases like left right or outer joins would bring additional information with two columns. Don't try to overengineer your merge line, be explicit as you suggest

Solution 1:

df2.columns = ['Col2', 'UserName']

pd.merge(df1, df2,on='UserName')
Out[67]: 
  Col1  UserName Col2
0    a         1    d
1    b         2    e
2    c         3    f

Solution 2:

pd.merge(df1, df2, left_on='UserName', right_on='UserID').drop('UserID', axis=1)
Out[71]: 
  Col1  UserName Col2
0    a         1    d
1    b         2    e
2    c         3    f
1
  • Note that Solution 2 is dangerous - this will not work in case df1 happens to also have a (possibly unrelated) UserID column. I have actually encountered this in real-life applications. In this case, Solution 2 actually throws an error since there will be no column named "UserID", only "UserID_x" and "UserID_y"!
    – Thomas
    Aug 3, 2021 at 10:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.