0

I have two dataframes: df1 and df2. They have the same shape. Here's what they look like:

df1

1 2 3 4 5
20% 10% 5% 1% 0%
20% 10% 5% 1% 0%
20% 10% 5% 1% 0%

df2

1 2 3 4 5
string word thing NaN NaN
string word thing NaN NaN
string word thing NaN NaN

I want to use df2 to mask or filter df1, such that "new_df1" looks like the below. In places where df2 has "NaN', I want df1 to have NaN. In places where df2 is not NaN, I want to keep the original value of df1.

new_df1

1 2 3 4 5
20% 10% 5% NaN NaN
20% 10% 5% NaN NaN
20% 10% 5% NaN NaN

I've found functions like df1.mask(df2=None), df1.mask(df2!=None), df1.where(df2!=None), and df1.where(df2=None). I either get a dataframe full of NaN or the opposite of what I want (NaN's in new_df1 where there is a value in df2). I suspect it's because the values in df1 are strings and not integers or floats? It looks like df.mask() and df.where() don't take string exceptions, and I can't find what the right function is, but it must exist. Can anyone help?

2 Answers 2

1

You can try:

new_df1 = df1.mask(df2.isnull())

Or you can also do:

new_df1 = df1.where(~df2.isnull())
8
  • Sadly, the first suggestion just gives me a dataframe full of NaN, and the second gives the opposite of I want. In other words, it's the same as the results I've gotten. :(
    – rcpi
    Commented Oct 18, 2022 at 0:28
  • It might be because of the difference of NaN and None, please try: new_df1 = df1.mask(pd.isnull(df2))
    – Isaac Rene
    Commented Oct 18, 2022 at 0:51
  • That also gives a df full of NaN. If I throw a ~ before the pd, then I get the opposite of what I want. I tried df1.where(pd.isnull(df2)) without and with the ~, and still the same two outputs. It's so frustratingly close. I don't think this should be necessary, but maybe I need to make an inverse of df2 (put NaN where I want to keep original values). Still, I can't image why I'd have to do it that way.
    – rcpi
    Commented Oct 18, 2022 at 0:54
  • Yes, it is strange, it works on my computer but I might not have the same data types. What do you get if you try print(df2.dtypes) and print(type(df2.iloc[0,4])
    – Isaac Rene
    Commented Oct 18, 2022 at 0:58
  • Really? Hm that's odd.
    – rcpi
    Commented Oct 18, 2022 at 1:00
1

Since the mask is aligned by index, you may need to reset the index of the two dataframes before doing the masking

Input

df1 = pd.DataFrame([[.1,.2], [.4,.5]], index=[1,2])
df2 = pd.DataFrame([['a',None], [None,'d']], index=[3,4])

Masking without aligning the index gives wrong result

df1.mask(df2.isnull())

    0   1
1   NaN NaN
2   NaN NaN

While resetting the index first gives correct result

df1.reset_index(drop=True).mask(df2.reset_index(drop=True).isnull())

    0   1
0   0.1 NaN
1   NaN 0.5
1
  • Wow that was it! It wasn't my index names, but there was one column header that was off. So I renumbered the columns, and now it works. Unbelievable. Thank you!!
    – rcpi
    Commented Oct 18, 2022 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.