14

I get this exception for a reason I do not understand. It is quite complicated, where my np.array v comes from, but here is the code when the exception occurs:

print v, type(v)

for val in v:
    print val, type(val)

print "use isfinte() with astype(float64): "
np.isfinite(v.astype("float64"))

print "use isfinite() as usual: "
try:
    np.isfinite(v)
except Exception,e:
    print e

This gives the following output:

[6.4441947744288255 7.2246449651781788 4.1028442021807656
 4.8832943929301189] <type 'numpy.ndarray'> 

6.44419477443 <type 'numpy.float64'>
7.22464496518 <type 'numpy.float64'>
4.10284420218 <type 'numpy.float64'>
4.88329439293 <type 'numpy.float64'>

np.isfinte() with astype(float64): 
[ True  True  True  True]

np.isfinte() as usual: 
ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

I do not understand the TypeError. All the elements are np.float64 and should be fine. Maybe a bug? This Error does only occure sometimes, but I can't find differences between the arrays. The always have the same type.

Thanks in advance.

EDIT: Working example:

Data Structures are as small as shown above.

import pandas as pd
import numpy as np


def forward_estim(H,end):

    old_idx = H.index
    new_idx = pd.period_range(old_idx[-1],end,freq=old_idx.freq)

    H_estim = pd.DataFrame(columns=["A","B","C","D"],index=new_idx)

    H_chg = H.values[1:]-H.values[:-1]
    mean_ = H_chg.mean()
    std_  = H_chg.std()

    H_estim.ix[0] = H.ix[-1]

    for i in range(1,len(H_estim)):
        H_estim.A[i] = H_estim.A[i-1] + mean_ + std_/2
        H_estim.B[i] = H_estim.B[i-1] + mean_ + std_
        H_estim.C[i] = H_estim.C[i-1] + mean_ - std_
        H_estim.D[i] = H_estim.D[i-1] + mean_ - std_/2

    return H_estim.ix[1:]


H_idx = pd.period_range("2010-01-01","2012-01-01",freq="A")
print H_idx

H = pd.Series(np.array([2.3,3.0,2.9]),index=H_idx)
print H

H_estim = forward_estim(H,"2014-01-01")
print H_estim

np.isfinite(H_estim.values.astype("float64"))
print "This works!"

np.isfinite(H_estim.values)
print "This does not work!"

This is run here using:

MacOsX Mavericks, Python 2.7.6, numpy 1.8.1, pandas 0.13.1

2
  • You should definitely state which versions of Python and numpy you are working with on which system. It would be of great benefit if you could come up with a minimal example reproducing the issue. May 22, 2014 at 13:39
  • Edited the original Post with an example. This will throw the exception mentioned above on my machine/python setup. May 22, 2014 at 14:51

2 Answers 2

13

H_estim.values is a numpy array with the data type object (take a look at H_estim.values.dtype):

In [62]: H_estim.values
Out[62]: 
array([[3.4000000000000004, 3.6000000000000005, 2.7999999999999998, 3.0],
       [3.9000000000000004, 4.3000000000000007, 2.6999999999999993,
        3.0999999999999996]], dtype=object)

In [63]: H_estim.values.dtype
Out[63]: dtype('O')

In an object array, the data stored in the array's memory are pointers to python objects, not the objects themselves. In this case, the objects are np.float64 instances:

In [65]: H_estim.values[0,0]
Out[65]: 3.4000000000000004

In [66]: type(H_estim.values[0,0])
Out[66]: numpy.float64

So in many respects, this array looks and acts like an array of np.float64 values, but it is not the same. In particular, the numpy ufuncs (including np.isfinite) don't handle object arrays.

H_estim.values.astype(np.float64) converts the array to one with data type np.float64 (i.e. an array where the array elements are the actual floating point values, not pointers to objects). Compare the following to the output shown above for H_estim.values.

In [70]: a = H_estim.values.astype(np.float64)

In [71]: a
Out[71]: 
array([[ 3.4,  3.6,  2.8,  3. ],
       [ 3.9,  4.3,  2.7,  3.1]])

In [72]: a.dtype
Out[72]: dtype('float64')
4
  • Thanks, Warren. This answers my question in much detail. To add up on my question: This is the second time "unexpected" behavior could be circumvented by using .astype(). May 22, 2014 at 22:55
  • ... So, do I do something conceptionally wrong here in the forward_estim() function or is it one of the drawbacks with dynamically typed languages that you tend to ignore/forget thinking about types (and "internal type structures" of packeges) and these kind of "enforcing" commands like .astype() are simply things you have to do sometimes. This function belongs to some 10.000s lines of code and so far things worked smoothly. May 22, 2014 at 23:02
  • 1
    The object array is created by Pandas. Pandas uses object arrays as a convenient way to handle arrays holding heterogeneous data types, and it usually works fine, but sometimes, yeah, you have to be aware of its limitations. By the way, if you are using isfinite to check for NaN, you could use the Pandas function isnull (pd.isnull, or the DataFrame method). But isnull doesn't help if you are checking for np.inf. May 22, 2014 at 23:43
  • I use DataFrame.dropna() a lot as the underlying database my algorithms are used on is big and quite ill May 23, 2014 at 0:32
1

You assume that "All the elements are np.float64 and should be fine.". However, this likely is not the case. How large is the data structure? Can you look at all values and find something suspicious? From http://matplotlib.1069221.n5.nabble.com/type-error-with-python-3-2-and-version-1-1-1-of-matplotlib-numpy-error-td38784.html we see that this problem might appear with Decimal data types. Is there a way for you to create a minimal working example that reproduces the issue? It should be possible, and when you create this example, it will most likely already pinpoint the issue.

1
  • Thanks for the link. I found that as well before asking. That is why I wrote the for loop printing the type() of the data to confirm they are np.float64. That is what confuses me, as using .astype("float64") will help although data is already of this type. May 22, 2014 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.