4

I have a data frame (df) that looks like

PID     SID     RID     
124     294     294
954     299     299
NAN     949     493
959     NAN     959
059     059     059 
0405    NAN     NAN
493     942     395

I used

testdf = df.eq(df["PID"], axis='index').all(axis=1) 

to get a list (testdf) that reports if the values across roles are equal, this works except that the NAN get in the way.

I tried to use

testdf = df.eq(df["PID"], axis='index').all(axis=1).notnull()

but for some reason it reports everything as equal when i know some rows aren't.

Here is an example of what I would want testdf to look like in the end

0    False
1    False
2    False
3    True
4    True
5    False
6    False
9
  • Your question is not very clear for me. You want to compare two DataFrame (df and `df2) with similar data ?
    – Romain
    Aug 29, 2015 at 21:17
  • Updated it now, hopefully its more clear.
    – r3vdev
    Aug 29, 2015 at 21:20
  • I don't understand why do you think the output is incorrect?
    – EdChum
    Aug 29, 2015 at 21:24
  • Ok, if I understand you want to get PID with SID==RID and SID!=RID. Is that correct ?
    – Romain
    Aug 29, 2015 at 21:26
  • EdChum, when i use testdf = df.eq(df["PID"], axis='index').all(axis=1).notnull() testdef reports everything as true, when I know some values across the rows are not equal.
    – r3vdev
    Aug 29, 2015 at 21:32

2 Answers 2

1

If NaN can be ignored we can fill NaN in each column (RID and SID) with each other values. If the remaining values are equal to the PID the result will be True else it will be False. You can do it on a copy of the DataFrame in order to not alter your original data.

df['SID'] = df['SID'].fillna(df['RID'])
df['RID'] = df['RID'].fillna(df['SID'])
testdf = df.eq(df['PID'], axis='index').all(axis=1)
testdf

Here is the result:

0    False
1    False
2    False
3     True
4     True
5    False
6    False
5
  • this would work if i was just comparing SID to RID but I want test across each row (index) if the values in SID PID and RID are the same but I want it to ignore NAN. so if index 3 would be true.
    – r3vdev
    Aug 29, 2015 at 21:50
  • Updated original post with an example of the output I am looking for
    – r3vdev
    Aug 29, 2015 at 21:54
  • It is more clear now. I have modified my answer, tell me if it can solve your problem.
    – Romain
    Aug 29, 2015 at 22:28
  • Cool even it was difficult for me to understand, sorry.
    – Romain
    Aug 29, 2015 at 22:39
  • Do not forget to check the answer to close the question if it works.
    – Romain
    Aug 29, 2015 at 22:52
1

This problem is caused by the fact that np.nan == np.nan is False and np.nan != np.nan is True. A quick workaround would be to replace any nan in df and df2 with something that you know is not in your dataframes eg foo:

df = df.fillna("foo")
df2 = df2.fillna("foo")

You can then compare your dataframes as you wish.

1
  • 2
    If I fill with foo wouldn't I still get a false when comparing across rows?
    – r3vdev
    Aug 29, 2015 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.