I am using sklearn.svm.svc from scikit-learn to do binary classification. I am using its predict_proba() function to get probability estimates. Can anyone tell me how predict_proba() internally calculates the probability?
2 Answers
Scikit-learn uses LibSVM internally, and this in turn uses Platt scaling, as detailed in this note by the LibSVM authors, to calibrate the SVM to produce probabilities in addition to class predictions.
Platt scaling requires first training the SVM as usual, then optimizing parameter vectors A and B such that
P(y|X) = 1 / (1 + exp(A * f(X) + B))
where f(X)
is the signed distance of a sample from the hyperplane (scikit-learn's decision_function
method). You may recognize the logistic sigmoid in this definition, the same function that logistic regression and neural nets use for turning decision functions into probability estimates.
Mind you: the B
parameter, the "intercept" or "bias" or whatever you like to call it, can cause predictions based on probability estimates from this model to be inconsistent with the ones you get from the SVM decision function f
. E.g. suppose that f(X) = 10
, then the prediction for X
is positive; but if B = -9.9
and A = 1
, then P(y|X) = .475
. I'm pulling these numbers out of thin air, but you've noticed that this can occur in practice.
Effectively, Platt scaling trains a probability model on top of the SVM's outputs under a cross-entropy loss function. To prevent this model from overfitting, it uses an internal five-fold cross validation, meaning that training SVMs with probability=True
can be quite a lot more expensive than a vanilla, non-probabilistic SVM.
-
2Great answer @larsmans. I'm just wondering if the probabilities can be interpreted as a confidence measure for the classification decisions? E.g. very close probabilities for positive and negative classes for a sample means the learner is less sure about its classification?– Moses XuFeb 28, 2013 at 1:20
-
2Thanks @larsmans. I've actually observed much more dramatic cases -- predictions of 1, but with probability 0.45. I thought that the bayes optimal cutoff used is 0.5 precisely. Do you reckon that such dramatic cases can still be explained by the numerical instability in LibSVM?– Moses XuMar 9, 2013 at 8:07
-
1@MosesXu: this is something worth investigating, but I don't have the time to dig into the LibSVM code ATM. It seems to be inconsistent behavior at first sight, but I think
predict
does not actually use the probabilities, but rather the SVM hyperplane.– Fred FooMar 9, 2013 at 13:46 -
2@MosesXu: I stared at the math a little longer and I realized that with an appropriate value of
B
, you can get predictions that are really different from the ones you get from the SVMpredict
anddecision_function
methods. I fear that when you use Platt scaling, you'll have to commit yourself to either believingpredict
, or believingpredict_proba
, as the two may be inconsistent.– Fred FooMar 9, 2013 at 13:50 -
1@larsmans: it is somewhat surprising that the predict function always sticks to the hyperplane regardless of the probability parameter -- is this because the learned hyperplane always represents minimum structural risk while the fitted logistic regression, though fitted using n-fold cross validation, is still prone to over fitting?– Moses XuMar 12, 2013 at 3:42
Actually I found a slightly different answer that they used this code to convert decision value to probability
'double fApB = decision_value*A+B;
if (fApB >= 0)
return Math.exp(-fApB)/(1.0+Math.exp(-fApB));
else
return 1.0/(1+Math.exp(fApB)) ;'
Here A and B values can be found in the model file (probA and probB). It offers a way to convert probability to decision value and thus to hinge loss.
Use that ln(0) = -200.