2

as I understand, support vector regression in Scikit learn takes an integer for the degree. However, it seems to me as if lower degree polynomials are not considered.

Running the following example:

import numpy
from sklearn.svm import SVR
X = np.sort(5 * np.random.rand(40, 1), axis=0)
Y=(2*X-.75*X**2).ravel()
Y[::5] += 3 * (0.5 - np.random.rand(8))
svr_poly = SVR(kernel='poly', C=1e3, degree=2)
y_poly = svr_poly.fit(X, Y).predict(X)

(as copied and slightly modified from here http://scikit-learn.org/stable/auto_examples/svm/plot_svm_regression.html)

Plotting the data gives a rather poor fit (even when skipping line 5 where a random error is given to the Y-values).

It seems like lower order terms are not considered. I tried to pass a list [1, 2] for the degree parameter but then I got an error for the predict command. Is there any way to include them? Did I miss something obvious?

3
  • You set the penalty C to 1000, it basically penalizes any effective parameters. Try setting it to 1 or less Oct 5, 2017 at 13:12
  • It helps, but it does not solve my general question to what extent lower order terms are considered. Oct 5, 2017 at 14:06
  • This comment stats.stackexchange.com/questions/152610/… is very useful, however I cannot infere from it if lower order terms are included or not Oct 5, 2017 at 15:48

2 Answers 2

2

I think the lower order polynomial terms are included in the fitted model but are not visible in the plot since the C and epsilon parameters are not well suited for the data. One can usually obtain a better fit by fine-tuning the parameters with GridSearchCV. Since in this case the data is not centered the coef0 parameter also has a significant effect.

The following parameters should give a better fit for the data:

svr_poly = SVR(kernel='poly', degree=2, C=100, epsilon=0.0001, coef0=5)
1
  • Thanks! I thought for a "simple" model, the hyperparameters would not be that important. I will accept the answere as it lead me to an example which shows more clearly that they are included. I will provide it as an answere, too. Oct 6, 2017 at 7:09
1

scikit-learn.SVR runs lower order polynomial. A modification of the original example shows this clearly.

X = np.sort(2*np.random.rand(40,1)-1,axis=0)
Y = np.sin(6*X).ravel()
svr_poly1 = SVR(kernel='poly', C=1e3, degree=3)
y_poly1 = svr_poly1.fit(X, Y).predict(X)
svr_poly2 = SVR(kernel='poly', C=100, epsilon=0.0001, coef0=5, degree=3)
y_poly2 = svr_poly2.fit(X, Y).predict(X)
svr_poly3 = SVR(kernel='poly', C=100, epsilon=0.0001, coef0=5, degree=5)
y_poly3 = svr_poly3.fit(X, Y).predict(X)

Plotting this gives

Result of the different SVR algorithms with two models using order 3 but different hyperparameters and one model using order 5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.