34

I want to use numpy.exp like this:

cc = np.array([
    [0.120,0.34,-1234.1]
])

print 1/(1+np.exp(-cc))

But this gives me error:

/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp

I can't understand why? How can I fix this? It seems the problem is with third number (-1234.1)

1

7 Answers 7

50

As fuglede says, the issue here is that np.float64 can't handle a number as large as exp(1234.1). Try using np.float128 instead:

>>> cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
>>> cc
array([[ 0.12,  0.34, -1234.1]], dtype=float128)
>>> 1 / (1 + np.exp(-cc))
array([[ 0.52996405,  0.58419052,  1.0893812e-536]], dtype=float128)

Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don't actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here.

For most practical purposes, you can probably approximate 1 / (1 + <a large number>) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):

>>> 1 / (1 + np.exp(-cc))
/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.4
array([[ 0.52996405,  0.58419052,  0.        ]])

If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:

>>> from scipy.special import expit
>>> expit(cc)
array([[ 0.52996405,  0.58419052,  0.        ]])
0
20

The largest value representable by a numpy float is 1.7976931348623157e+308, whose logarithm is about 709.782, so there is no way to represent np.exp(1234.1).

In [1]: import numpy as np

In [2]: np.finfo('d').max
Out[2]: 1.7976931348623157e+308

In [3]: np.log(_)
Out[3]: 709.78271289338397

In [4]: np.exp(709)
Out[4]: 8.2184074615549724e+307

In [5]: np.exp(710)
/usr/local/bin/ipython:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.5
Out[5]: inf
1
  • 1
    BTW, exp(1234.1) is equal to 9.17952305220754e+535.
    – ForceBru
    Nov 21, 2016 at 18:09
6

A possible solution is to use the decimal module, which lets you work with arbitrary precision floats. Here is an example where a numpy array of floats with 100 digits precision is used:

import numpy as np
import decimal

# Precision to use
decimal.getcontext().prec = 100

# Original array
cc = np.array(
    [0.120,0.34,-1234.1]
)
# Fails
print(1/(1 + np.exp(-cc)))    

# New array with the specified precision
ccd = np.asarray([decimal.Decimal(el) for el in cc], dtype=object)
# Works!
print(1/(1 + np.exp(-ccd)))
2
  • 1
    Do you lose the advantage of vectorized operations when working with decimal? I'd guess that since numpy would have to treat it as an object, you might implicitly lose speed?
    – Praveen
    Nov 24, 2016 at 23:54
  • 2
    You loose a lot of speed indeed. Actually, numpy delegates the entire computation to the Decimal objects, operations on which cannot be performed natively by your CPU. For a large array the time difference will be severe.
    – jmd_dk
    Nov 25, 2016 at 7:48
3

exp(-1234.1) is too small for 32bit or 64bit floating-point numbers. Since it cannot be represented, numpy produces the correct warning.

Using IEEE 754 32bit floating-point numbers, the smallest positive number it can represent is 2^(-149), which is roughly 1e-45.

If you use IEEE 754 64 bit floating-point numbers, the smallest positive number is 2^(-1074) which is roughy 1e-327.

In either case, it cannot represent a number as small as exp(-1234.1) which is about 1e-535.

You should be using the expit function from scipy to compute the sigmoid function. This would give you better precision.

For practical purposes, exp(-1234.1) is a very small number. If rounding to zero makes sense in your use case, numpy produces benign results by rounding it to zero.

1

If you don’t care about precision, you can use numpy.clip.

In float64:

cc = np.clip(cc, -709.78, 709.78)

In float32:

cc = np.clip(cc, -88.72, 88.72)
0

As mentioned earlier by Praveen, you can use expit from scipy

So the problem can be solved by using: 1 / (1+ exp(-x)) = exp(x) / (1+exp(x))

>>> import numpy as np
>>> cc = np.array([[0.120,0.34,-1234.1]])
>>> np.exp(cc) / (1 +  np.exp(cc))
array([[0.52996405, 0.58419052, 0.        ]])
0

This overflow can be handled mathematically; you may not need to convert your values to np.float128

A sigmoid function is defined by:

enter image description here

Now, when you rearrange this function by multiplying numerator and denominator by e**x.

You will get:

enter image description here

So, you can design your sigmoid function accordingly:

import numpy as np

def sigmoid(x):
  if x > 0:   
    z = np.exp(-x)
    return 1/(1+z)
  else:
    z = np.exp(x)
    return z/(1+z)

Coming back to the question:

cc = np.array([0.120,0.34,-1234.1])
print([sigmoid(x) for x in cc])

#output
[0.5299640517645717, 0.5841905229354074, 0.0]

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.