2

I am trying to do a left outer join on two Data.Frames using Python. The goal is to get a column from right into left based on if the key from left exists in the list from right.

My initial thought was to use Pandas, so I wrote something like this:

import pandas as pd

left = pd.DataFrame({'name':['spam', 'ham', 'eggs'], 'leftkey':[11, 22, 33]})
right = pd.DataFrame({'var':['foo', 'bar'], 'rightkey':[[1, 2, 5], [2, 33, 100]]})

merged = pandas.merge(left, right, left_on='keyleft', right_on='keyright', how='left')

As we can see, left_on is a single variable, while right_on is a list.

I would expect merged to look something like this:

|   | name | leftkey | var | rightkey   |
|---|------|---------|-----|------------|
| 0 | spam | 11      | NaN | NaN        |
| 1 | ham  | 22      | NaN | NaN        |
| 2 | eggs | 33      | bar | [2,33,100] |

However, all of var and rightkey end up being NaN.

I realize that I could just put everything in R and have this done. Perhaps I'm overthinking things and this does not even require Pandas. However, my hope is to keep the pipeline in Python for as long as possible.

Any suggestions?

1 Answer 1

1

You can use the method explode to build a new column that you can use as the right_on argument in merge:

right = right.assign(rightkey_x = right['rightkey']).explode('rightkey_x')

Output:

   var      rightkey rightkey_x
0  foo     [1, 2, 5]          1
0  foo     [1, 2, 5]          2
0  foo     [1, 2, 5]          5
1  bar  [2, 33, 100]          2
1  bar  [2, 33, 100]         33
1  bar  [2, 33, 100]        100

Then you can merge both dataframes and drop the helper column:

pd.merge(left, right, left_on='leftkey', right_on='rightkey_x', how='left')\
.drop('rightkey_x', axis=1)

Output:

   name leftkey  var      rightkey
0  spam      11  NaN           NaN
1   ham      22  NaN           NaN
2  eggs      33  bar  [2, 33, 100]
2
  • This solution may end up being the option I go with, but not accepting it just yet. My concerns are 1) that this will create a decent amount of slow-down when applied to 100k+ entries, and 2) that there is a more aesthetically pleasing solution. That being said, thank you! I will give this a go shortly and see if it at least gets me to the next bend in the pipeline.
    – Salt
    Feb 11, 2020 at 17:54
  • Of course. Unfortunately, Pandas doesn't always has an aesthetically pleasing solution like R. In any case first you need to extract those values from lists. Feb 11, 2020 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.