17

I found sklearn.svm.LinearSVC and sklearn.svm.SVC(kernel='linear') and they seem very similar to me, but I get very different results on Reuters.

sklearn.svm.LinearSVC: 81.05% in   28.87s train /    9.71s test
sklearn.svm.SVC      : 33.55% in 6536.53s train / 2418.62s test

Both have a linear kernel. The tolerance of the LinearSVC is higher than the one of SVC:

LinearSVC(C=1.0, tol=0.0001, max_iter=1000, penalty='l2', loss='squared_hinge', dual=True, multi_class='ovr', fit_intercept=True, intercept_scaling=1)
SVC      (C=1.0, tol=0.001,    max_iter=-1, shrinking=True, probability=False, cache_size=200, decision_function_shape=None)

How do both functions differ otherwise? Even if I set kernel='linear, tol=0.0001, max_iter=1000 anddecision_function_shape='ovr'theSVCtakes much longer thanLinearSVC`. Why?

I use sklearn 0.18 and both are wrapped in the OneVsRestClassifier. I'm not sure if this makes the same as multi_class='ovr' / decision_function_shape='ovr'.

4
  • Can you upgrade to 0.18.2 and see if the results still differ ?
    – seralouk
    Jul 29, 2017 at 10:23
  • I believe the version is not the case. The sklearn documentation contains the examples of fitting these classifiers. The results differ due to approaches that the models use.
    – E.Z
    Jul 29, 2017 at 13:32
  • There are some discussions about it already, maybe check these: stackoverflow.com/questions/33843981/… and stackoverflow.com/questions/35076586/…
    – phev8
    Jul 29, 2017 at 14:05
  • The documentation states that they use different implementations and even using the method from sklearn or accessing directly the low-level implementation can result in different scores.
    – phev8
    Jul 29, 2017 at 14:11

2 Answers 2

27

Truly, LinearSVC and SVC(kernel='linear') yield different results, i. e. metrics score and decision boundaries, because they use different approaches. The toy example below proves it:

from sklearn.datasets import load_iris
from sklearn.svm import LinearSVC, SVC

X, y = load_iris(return_X_y=True)

clf_1 = LinearSVC().fit(X, y)  # possible to state loss='hinge'
clf_2 = SVC(kernel='linear').fit(X, y)

score_1 = clf_1.score(X, y)
score_2 = clf_2.score(X, y)

print('LinearSVC score %s' % score_1)
print('SVC score %s' % score_2)

--------------------------
>>>    0.96666666666666667
>>>    0.98666666666666669

The key principles of that difference are the following:

  • By default scaling, LinearSVC minimizes the squared hinge loss while SVC minimizes the regular hinge loss. It is possible to manually define a 'hinge' string for loss parameter in LinearSVC.
  • LinearSVC uses the One-vs-All (also known as One-vs-Rest) multiclass reduction while SVC uses the One-vs-One multiclass reduction. It is also noted here. Also, for multi-class classification problem SVC fits N * (N - 1) / 2 models where N is the amount of classes. LinearSVC, by contrast, simply fits N models. If the classification problem is binary, then only one model is fit in both scenarios. multi_class and decision_function_shape parameters have nothing in common. The second one is an aggregator that transforms the results of the decision function in a convenient shape of (n_features, n_samples). multi_class is an algorithmic approach to establish a solution.
  • The underlying estimators for LinearSVC are liblinear, that do in fact penalize the intercept. SVC uses libsvm estimators that do not. liblinear estimators are optimized for a linear (special) case and thus converge faster on big amounts of data than libsvm. That is why LinearSVC takes less time to solve the problem.

In fact, LinearSVC is not actually linear after the intercept scaling as it was stated in the comments section.

1
-1

The main difference between them is linearsvc lets your choose only linear classifier whereas svc let yo choose from a variety of non-linear classifiers. however it is not recommended to use svc for non-linear problems as they are super slow. try importing other libraries for doing non-linear classifications.

now the point that even after defining kernel='linear' we don't get same output is because both linearsvc and svc try different approaches while doing the background mathematics. also linearsvc works on principle of one-vs-rest, and svc works on one-vs-one.

I hope this answers your question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.