I'm working with numpy arrays of different data types. I would like to know, of any particular array, which elements are NaN. Normally, this is what np.isnan
is for.
However, np.isnan
isn't friendly to arrays of data type object
(or any string data type):
>>> str_arr = np.array(["A", "B", "C"])
>>> np.isnan(str_arr)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Not implemented for this type
>>> obj_arr = np.array([1, 2, "A"], dtype=object)
>>> np.isnan(obj_arr)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
What I would like to get out of these two calls is simply np.array([False, False, False])
. I can't just put try
and except TypeError
around my call to np.isnan
and assume that any array that generates a TypeError
does not contain NaNs: after all, I'd like np.isnan(np.array([1, np.NaN, "A"]))
to return np.array([False, True, False])
.
My current solution is to make a new array, of type np.float64
, loop through the elements of the original array, try
ing to put that element in the new array (and if it fails, leave it as zero) and then calling np.isnan
on the new array. However, this is of course rather slow. (At least, for large object arrays.)
def isnan(arr):
if isinstance(arr, np.ndarray) and (arr.dtype == object):
# Create a new array of dtype float64, fill it with the same values as the input array (where possible), and
# then call np.isnan on the new array. This way, np.isnan is only called once. (Much faster than calling it on
# every element in the input array.)
new_arr = np.zeros((len(arr),), dtype=np.float64)
for idx in xrange(len(arr)):
try:
new_arr[idx] = arr[idx]
except Exception:
pass
return np.isnan(new_arr)
else:
try:
return np.isnan(arr)
except TypeError:
return False
This particular implementation also only works for one-dimensional arrays, and I can't think of a decent way to make the for
loop run over an arbitrary number of dimensions.
Is there a more efficient way to figure out which elements in an object
-type array are NaN?
EDIT: I'm running Python 2.7.10.
Note that [x is np.nan for x in np.array([np.nan])]
returns False
: np.nan
is not always the same object in memory as a different np.nan
.
I do not want the string "nan"
to be considered equivalent to np.nan
: I want isnan(np.array(["nan"], dtype=object))
to return np.array([False])
.
The multi-dimensionality isn't a big issue. (It's nothing that a little ravel
-and-reshape
ing won't fix. :p)
Any function that relies on the is
operator to test equivalence of two NaNs isn't always going to work. (If you think they should, ask yourself what the is
operator actually does!)