132

I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe.

For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array.

What's going on?!

set([type(x) for x in tester])
Out[59]: {float}

tester
Out[60]: 
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan], dtype=object)

set([type(x) for x in tester])
Out[61]: {float}

np.isnan(tester)
Traceback (most recent call last):

File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

set([np.isnan(x) for x in tester])
Out[65]: {False, True}

type(tester)
Out[66]: numpy.ndarray

5 Answers 5

206

np.isnan can be applied to NumPy arrays of native dtype (such as np.float64):

In [99]: np.isnan(np.array([np.nan, 0], dtype=np.float64))
Out[99]: array([ True, False], dtype=bool)

but raises TypeError when applied to object arrays:

In [96]: np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Since you have Pandas, you could use pd.isnull instead -- it can accept NumPy arrays of object or native dtypes:

In [97]: pd.isnull(np.array([np.nan, 0], dtype=float))
Out[97]: array([ True, False], dtype=bool)

In [98]: pd.isnull(np.array([np.nan, 0], dtype=object))
Out[98]: array([ True, False], dtype=bool)

Note that None is also considered a null value in object arrays.

1
  • 4
    Thanks - used pd.isnull(). Does not appear to be any performance impact either.
    – tim654321
    Commented Mar 15, 2016 at 4:50
14

A great substitute for np.isnan() and pd.isnull() is

for i in range(0,a.shape[0]):
    if(a[i]!=a[i]):
       //do something here
       //a[i] is nan

since only nan is not equal to itself.

7
  • that may not work for arrays because it raises the well-known "ValueError: Truth value of a xxx is ambiguous".
    – MSeifert
    Commented Dec 19, 2016 at 4:42
  • @MSeifert Are you talking about python? I just use this method to do something in machine learning, Why I didn't encounter the well-known error?
    – Statham
    Commented Dec 19, 2016 at 7:58
  • Yes, seems like you haven't used numpy or pandas before. Just use import numpy as np; a = np.array([1,2,3, np.nan]) and run your code.
    – MSeifert
    Commented Dec 19, 2016 at 8:18
  • @MSeifert er, I am new to numpy but the code ran ok, no error happed
    – Statham
    Commented Dec 19, 2016 at 8:33
  • In [1]: import numpy as np In [2]: a=np.array([1,2,3,np.nan]) In [3]: print a [ 1. 2. 3. nan] In [4]: print a[3]==a[3] False
    – Statham
    Commented Dec 19, 2016 at 8:33
10

On top of @unutbu answer, you could coerce pandas numpy object array to native (float64) type, something along the line

import pandas as pd
pd.to_numeric(df['tester'], errors='coerce')

Specify errors='coerce' to force strings that can't be parsed to a numeric value to become NaN. Column type would be dtype: float64, and then isnan check should work

2
  • His name seems to be unutbu ;)
    – Dr_Zaszuś
    Commented May 26, 2020 at 11:06
  • @Dr_Zaszuś Thanks, fixed Commented May 26, 2020 at 15:48
1

Make sure you import csv file using Pandas

import pandas as pd

condition = pd.isnull(data[i][j])
0

Just answer this for a reminder of myself. It took me a whole day to solve. After digging deep into the code, I found that in _encodepy.py:

if values.dtype.kind in 'UO':
    # correct branch
else
    # wrong branch, if in this branch whatever data you give it will produce the error
    if np.isnan(known_values).any(): # here is problematic line

so the solution is very simple, just astype your data with np.object

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.