35

I need to remove all rows in which elements from column 3 onwards are all NaN

df = DataFrame(np.random.randn(6, 5), index=['a', 'c', 'e', 'f', 'g','h'], columns=['one', 'two', 'three', 'four', 'five'])

df2 = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
df2.ix[1][0] = 111
df2.ix[1][1] = 222

In the example above, my final data frame would not be having rows 'b' and 'c'.

How to use df.dropna() in this case?

0

1 Answer 1

76

You can call dropna with arguments subset and how:

df2.dropna(subset=['three', 'four', 'five'], how='all')

As the names suggests:

  • how='all' requires every column (of subset) in the row to be NaN in order to be dropped, as opposed to the default 'any'.
  • subset is those columns to inspect for NaNs.

As @PaulH points out, we can generalise to drop the last k columns with:

subset=df2.columns[k:]

Indeed, we could even do something more complicated if desired:

subset=filter(lambda x: len(x) > 3, df2.columns)
2
  • 13
    perhaps df2.dropna(subset=df2.columns[k:], how='all')? (just to be more general)
    – Paul H
    Commented Feb 21, 2013 at 2:00
  • @Thanks Paul. This is what I was looking for Commented Feb 21, 2013 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.