0

How do you get a true outer join in pandas? Meaning that it actually gives you the entire output instead of combining the columns to merge on. This is kind of silly in my opinion because it then makes it hard to determine what kind of operation to do on a row. I do this all the time to detect whether I should Insert, Update, or Delete data, however I always have to create extra copies of the merge on columns, which is just a bunch of overhead (sometimes massive amounts) on certain data sets.

example:

import pandas as pd

keys = ["A","B"]

df1 = pd.DataFrame({"A":[1,2,3],"B":["one","two","three"],"C":["testThis","testThat", "testThis"],"D":[None,hash("B"),hash("C")]})
df2 = pd.DataFrame({"A":[2,3,4],"B":["two","three","four"],"C":["testThis","testThat", "testThis"], "D":[hash("G"),hash("C"),hash("D")]})

fullJoinDf = df1.merge(df2, how="outer", left_on=keys, right_on=keys, suffixes=["","_r"])
display(
    fullJoinDf,
)

    A   B       C           D               C_r          D_r
0   1   one     testThis    NaN             NaN          NaN
1   2   two     testThat    -3.656526e+18   testThis    -9.136326e+18
2   3   three   testThis    -8.571400e+18   testThat    -8.571400e+18
3   4   four    NaN         NaN             testThis    -4.190116e+17

Noticed how it output A & B magically combined to a single set of columns. What I want is what I would get in SQL outerjoins etc like:

    A    B      C           D               A_r  B_r     C_r        D_r
0   1    one    testThis    NaN             NaN  NaN     NaN        NaN     
1   2    two    testThat    -3.656526e+18   2    two     testThis   -9.136326e+18
2   3    three  testThis    -8.571400e+18   3    three   testThat   -8.571400e+18
3   NaN  NaN    NaN         NaN             4    four    testThis   -4.190116e+17

Edit for @Felipe Whitaker

Using concat:

df3 = df1.copy().set_index(keys)
df4 = df2.copy().set_index(keys)
t = pd.concat([df3,df4], axis=1)
t.reset_index(), 

    A   B       C           D               C           D
0   1   one     testThis    NaN             NaN         NaN
1   2   two     testThat    -3.656526e+18   testThis    -9.136326e+18
2   3   three   testThis    -8.571400e+18   testThat    -8.571400e+18
3   4   four    NaN         NaN             testThis    -4.190116e+17

EDIT Examples* Given the answers I'm posting more tests, so anyone else who stumbles on this can see a few more of the "gatcha" variations I've discovered while doing this.

import pandas as pd

keys = ["A","B"]

df1 = pd.DataFrame({"A":[1,2,3],"B":["one","two","three"],"C":["testThis","testThat", "testThis"],"D":[None,hash("B"),hash("C")]})
df2 = pd.DataFrame({"A":[2,3,4],"B":["two","three","four"],"C":["testThis","testThat", "testThis"], "D":[hash("G"),hash("C"),hash("D")]})

df3 = df1.copy()
df4 = df2.copy()
df3.index = df3[keys]
df4.index = df4[keys]

df5 = df1.copy().set_index(keys)
df6 = df2.copy().set_index(keys)


fullJoinDf = df5.merge(df6, how="outer", left_on=keys, right_on=keys, suffixes=["","_r"])
fullJoinDf_2 = df3.merge(df4, how="outer", left_index=True, right_index=True, suffixes=["","_r"])
t = pd.concat([df1,df2], axis=1, keys=["A","B"])
display(
    df3.index,
    df5.index,
    fullJoinDf,
    fullJoinDf_2,
    t,
)

Index([(1, 'one'), (2, 'two'), (3, 'three')], dtype='object')
MultiIndex([(1,   'one'),
            (2,   'two'),
            (3, 'three')],
           names=['A', 'B'])

    A   B       C           D               C_r         D_r
0   1   one     testThis    NaN             NaN         NaN
1   2   two     testThat    -3.656526e+18   testThis    -9.136326e+18
2   3   three   testThis    -8.571400e+18   testThat    -8.571400e+18
3   4   four    NaN         NaN             testThis    -4.190116e+17

            A    B      C           D               A_r  B_r    C_r        D_r
(1, one)    1.0  one    testThis    NaN             NaN  NaN    NaN        NaN
(2, two)    2.0  two    testThat    -3.656526e+18   2.0  two    testThis    -9.136326e+18
(3, three)  3.0  three  testThis    -8.571400e+18   3.0  three  testThat    -8.571400e+18
(4, four)   NaN  NaN    NaN         NaN             4.0  four   testThis    -4.190116e+17

    A   B       C           D               A   B       C           D
0   1   one     testThis    NaN             2   two     testThis    -9136325526401183790
1   2   two     testThat    -3.656526e+18   3   three   testThat    -8571400026927442160
2   3   three   testThis    -8.571400e+18   4   four    testThis    -419011572131270498
14
  • Why don't you use pd.concat(iter, axis = 1)? Oct 27, 2020 at 23:29
  • @FelipeWhitaker - concat seems to do the same thing, see edit. Oct 27, 2020 at 23:38
  • 1
    I truly though it would just concatenate them. This result is counter intuitive. Well, thanks. Oct 27, 2020 at 23:48
  • Aren't you asking for a column-wise concat, not an outer join (it's only a join if there's at least common column, right?)? I can't understand your example, please edit to clarify what you mean by "This is kind of silly in my opinion because it then makes it hard to determine what kind of operation to do on a row."? Isn't that why we usually design schemas to have some primary key(/id) across database tables? If not, how can we make sense of your data?
    – smci
    Oct 28, 2020 at 0:12
  • I don't see what your schema gains from having two columns "A":[1,2,3],"B":["one","two","three"] with different incompatible id's, it just looks like a bad schema design to me if you intend it to work with joins and merges.
    – smci
    Oct 28, 2020 at 0:17

2 Answers 2

3

If you don't care abut the original indexes at all:

df1.index = df1[keys]
df2.index = df2[keys]

fullJoinDf = df1.merge(df2, how="outer", left_index=True, right_index=True, suffixes=["","_r"])

Result:

     A      B         C             D  A_r    B_r       C_r           D_r
0  1.0    one  testThis           NaN  NaN    NaN       NaN           NaN
1  2.0    two  testThat  6.368540e+18  2.0    two  testThis -6.457388e+18
2  3.0  three  testThis -7.490461e+18  3.0  three  testThat -7.490461e+18
3  NaN    NaN       NaN           NaN  4.0   four  testThis  4.344649e+18
5
  • df1.index = df1[keys] is just a disguised df1.set_index(['A','B'], inplace=True, drop=False) which is what should have been done on the dataframe in the first place.
    – smci
    Oct 28, 2020 at 3:15
  • @smci - your comment is incorrect. df1.index = df1[keys], returns a different result from df1.set_index(['A','B']). This is the entire crux of the problem and why all my examples above don't work, but this answer does. Nov 5, 2020 at 18:30
  • @JamieMarshall: this is using a list-of-tuples as index instead of a pandas MultiIndex. Really it seems like an XY problem to kludge the equivalence of the index columns 'A' and 'B', in a non-pandas way. This doesn't seem like a good idiom, and can you guarantee pandas can handle or won't mangle this dataframe in any further operations? (join, merge, reset_index etc. reset_index() doesn't work, for one)
    – smci
    Nov 10, 2020 at 21:08
  • @smci - bottom line, set_index doesn't work. I don't see how there's a way around this unless you can keep set_index from eating up the index columns. I can't. Nov 10, 2020 at 22:45
  • @JamieMarshall: I already showed you that df1.set_index(['A','B'], inplace=True, drop=False) prevents pandas eating up the index columns. You missed the crucial drop=False part. Also inplace=True to prevent unwanted copying. I think this is what you wanted; if not please explain specifically why not.
    – smci
    Nov 10, 2020 at 23:41
2

if you rename the columns used in the merge in 1 of the DataFrames before the merge it looks like it will give the correct answer

df1.merge(df2.rename({'A': 'A_y', 'B': 'B_y'}, axis =1), left_on=keys, right_on=['A_y', 'B_y'], how='outer')
#output:
    A   B       C_x         D_x             A_y     B_y     C_y         D_y
0   1.0 one     testThis    NaN             NaN     NaN     NaN         NaN
1   2.0 two     testThat    -2.482945e+18   2.0     two     testThis    -1.215774e+18
2   3.0 three   testThis    1.140152e+17    3.0     three   testThat    1.140152e+17
3   NaN NaN     NaN         NaN             4.0     four    testThis    -4.915382e+18
1
  • 1
    I marked @CodeDifferent right, but this maybe the faster operation. Setting the index requires the hole column, to be reset, this may be the more efficient Will have to test to find out. Oct 28, 2020 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.