1

I have a lists in one column of a dataframe df1, and I want to check to see for each row if all elements of that list are in another column that is in a second dataframe df2.

The two dataframes are something like this:

df1                                          df2

id | members      |                          num  |  available           |
1  |['a',b']      |                          one  | ['a','b','c','d','e']|
2  |['b']         |                          two  | ['a','b']            |
3  |['a','b','c'] |                          three| ['b','d','e']        |

I am trying to come up with a method that can give me which rows in df2 have all elements of members for each row in df1. Maybe something that yields this:


id | members      | which_cols            |                
1  |['a',b']      | ['one','two']         |                       
2  |['b']         | ['one','two','three'] |                         
3  |['a','b','c'] | ['one']               |                      

I thought converting it into dictionaries like {k: list(v) for k,v in df1.groupby("id")["members"]} and {i: list(j) for i,j in df2.groupby("num")["available"]} might make it more flexible to achieve the desired output but still haven't found a method to get to what I'm looking for.

df2 will have about 300 rows with length of available being as large as 25,000. And df1 can be as big as 1M rows with list length in members up to 15. So I think efficiency will also be important.

4
  • 1
    c does not appear in two? Feb 28, 2020 at 19:09
  • Just added some more info on the data. Feb 28, 2020 at 19:15
  • It's interesting that your members are a lot more abundant than available. You can make your problem a little easier by filtering out all members lists with length more than 15 as they wouldn't fit in any of available. Feb 28, 2020 at 19:23
  • that was another mistake on my part. It is the other way around. available will be like a master list and members are subsets but they are not all in all rows of df2 Feb 28, 2020 at 19:30

1 Answer 1

0

The core of the problem lies in your data setup. If you do a bit of preprocessing, you can avoid tediously iterating through every list multiple times over.

Setup

df1 = pd.Series([['a', 'b'], ['b'], ['a', 'b', 'c']], name = 'members').to_frame()
df2 = pd.Series([['a', 'b', 'c', 'd', 'e'], 
                  ['a', 'b'],
                  ['b', 'd', 'e']], name = 'available').to_frame()
df2.index = ['one', 'two', 'three']

>>> df1

    members
0   ['a', 'b']
1   ['b']
2   ['a', 'b', 'c']

>>> df2

        available
one.    ['a', 'b', 'c', 'd', 'e']
two     ['a', 'b']
three   ['b', 'd', 'e']

Reshape Data

If you one-hot encode your data before working with it, you put yourself at a great advantage for doing subset checks:

# You can do this many ways, but sklearn makes this very easy with:
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()
df1 = df1.join(pd.DataFrame(mlb.fit_transform(df1.pop('members')),
                          columns=mlb.classes_, index=df1.index))

mlb = MultiLabelBinarizer()
df2 = df2.join(pd.DataFrame(mlb.fit_transform(df2.pop('available')),
                          columns=mlb.classes_, index=df2.index))

>>> df1
    a   b   c
0   1   1   0
1   0   1   0
2   1   1   1


>>> df2
        a   b   c   d   e
one     1   1   1   1   1
two     1   1   0   0   0
three   0   1   0   1   1

Calculation

The clever thing about this data format, is that now you can subtract df1 from df2 and if none of your resultant values are -1 (indicating a lack of an element in df2, then you add that to the list. Think of this as overlaying the two dataframes (aligning each resource) and then subtracting. And of course, this can be vectorized:

>>> df1.apply(lambda row: df2.index[((df2[df1.columns] - row) >= 0).all(axis = 1)], axis = 1)

0   Index(['one', 'two'], dtype='object')
1   Index(['one', 'two', 'three'], dtype='object')
2   Index(['one'], dtype='object')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.