0

I have a set of data which correspond to ages (in steps of 0.1) along the x axis, and probabilities along the y axis. I'm trying to interpolate the data so I can find the maximum and a range of ages which covers 95% of the probability.

I've tried a simple interpolation using the code below, taken from the SciPy help pages, and it produces good results (I change the x and y variables to read my data), except for one feature.

from scipy.interpolate import interp1d

x = np.linspace(72, 100, num=29, endpoint=True)
y = df.iloc[:,0].values
f = interp1d(x, y)
f2 = interp1d(x, y, kind='cubic')

xnew = np.linspace(0, 10, num=41, endpoint=True)
import matplotlib.pyplot as plt
plt.plot(x, y, 'o', xnew, f(xnew), '-', xnew, f2(xnew), '--')
plt.legend(['data', 'linear', 'cubic'], loc='best')
plt.show()

The problem is, the cubic function works best, with the smoothest fit. However, it gives negative values for some parts of the probability curve, which is obviously not acceptable. Is there some way of setting a floor at y=0? I thought maybe switching to a quadratic kind would fix it, but it doesn't seem to. The linear fit does, but it's not smoothed, so is not a very good match.

I'm also not sure how to perform the second part of what I'm trying to do. It's probably very simple, but I don't know how to find the mean when I don't have a frequency table, but a grid of interpolated points which form a function. If I knew the function, I could integrate it, but I'm not sure how to do that in Python.

EDIT to include some data:

This is what my y data looks like:

array([3.41528917e-08, 7.81041275e-05, 9.60711716e-04, 5.75868934e-05,
       6.50260297e-05, 2.95556411e-05, 2.37331370e-05, 9.11990619e-05,
       1.08003254e-04, 4.16800419e-05, 6.63673113e-05, 2.57934035e-04,
       3.42235937e-03, 5.07534495e-03, 1.76603165e-02, 1.69535370e-01,
       2.67624254e-01, 4.29420872e-01, 8.25165926e-02, 2.08367339e-02,
       2.01227453e-03, 1.15405995e-04, 5.40163098e-07, 1.66905537e-10,
       8.31862858e-18, 4.14093219e-23, 8.32103362e-29, 5.65637769e-34,
       7.93547444e-40])
6
  • 1
    Please provide the sample data. I'm having trouble believing that cubic is best for something involving probabilities Commented Jul 5, 2019 at 13:05
  • I've added some data. When I say best, I mean that it looks like a smooth function.
    – Tom
    Commented Jul 5, 2019 at 13:09
  • 2
    Literally any continuos function will look like a smooth function. Have you tried a gaussian fit? Commented Jul 5, 2019 at 13:11
  • 2
    I know nothing about python, but your problem may be statistical / analytical rather than programmatical. Cubic functions aren't guaranteed to be positive in any given region and are guaranteed to be negative somewhere.
    – JDL
    Commented Jul 5, 2019 at 13:11
  • 1
    Another alternative to consider is a monotone interpolator such as SciPy's PchipInterpolator; see, for example, stackoverflow.com/questions/12935098/… Commented Jul 5, 2019 at 15:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.