5

I am having trouble understanding a function from sklearn and would like some clarification. At first I thought that sklearn's SVM's predict_proba function gave out the level of confidence of the classifier's prediction, but after playing around with it with my emotion recognition program, I am starting to form doubts and feel like I misunderstood the use and how the predict_proba function worked.

For example, I have my code setup something like this:

# Just finished training and now is splitting data (cross validation)
# and will give an accuracy after testing the accuracy of the test data

features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(main, target, test_size = 0.4)

model = SVC(probability=True)
model.fit(features_train, labels_train)
pred = model.predict(features_test)

accuracy = accuracy_score(labels_test, pred)
print accuracy

# Code that records video of 17 frames and forms matrix know as
# sub_main with features that would be fed into SVM

# Few lines of code later. . .  

model.predict(sub_main)
prob = model.predict_proba(sub_main)

prob_s = np.around(prob, decimals=5)
prob_s = prob_s* 100
pred = model.predict(sub_main)

print ''
print 'Prediction: '
print pred
print 'Probability: '
print 'Neutral: ', prob_s[0,0]
print 'Smiling: ', prob_s[0,1]
print 'Shocked: ', prob_s[0,2]
print 'Angry: ', prob_s[0,3]
print ''

And when I test it out, it gives me something like this:

Prediction: 
['Neutral']
Probability: 
Neutral:  66.084
Smiling:  17.875
Shocked:  11.883
Angry:  4.157

It managed to have a 66 percent confidence that the correct classification is "Neutral". 66 was next to "Neutral" which happened to be the highest number. The highest number was labeled with the actual prediction and I was happy about that.

But eventually eventually. . .

Prediction: 
['Angry']
Probability: 
Neutral:  99.309
Smiling:  0.16
Shocked:  0.511
Angry:  0.02

It made the prediction, "Angry" (which is the correct classification btw) and it assigned a confidence level of 99.3 percent next to "Neutral". The highest level of confidence (highest number) was assigned to Neutral despite the prediction being completely different.

Somtimes it also does this:

Prediction: 
['Smiling']
Probability: 
Neutral:  0.0
Smiling:  0.011
Shocked:  0.098
Angry:  99.891

Prediction: 
['Angry']
Probability: 
Neutral:  99.982
Smiling:  0.0
Shocked:  0.016
Angry:  0.001

I don't think understand how SVM's predict_proba function works and would like some clarification on how it works and what is going on with my code. What is going on in my code?

5
  • From the docs on SVC: "The probability model is created using cross validation, so the results can be slightly different than those obtained by predict. Also, it will produce meaningless results on very small datasets." How big is your training set?
    – Ryan
    Jul 1, 2015 at 4:18
  • Around 550 examples. Is that considered too small for the predict_proba function? Jul 1, 2015 at 5:47
  • @user3377126 The sample size looks fine. What's the accuracy score of your training set?
    – Jianxun Li
    Jul 1, 2015 at 6:11
  • It's a range of 80 to 90 percent with cross validation Jul 1, 2015 at 6:53
  • Have you tried setting class_weight='auto' in your SVC? From what I understand, the predict_proba uses Platt scaling which uses cross-validation. If you have some classes which are grossly outnumbered in the data set, I wouldn't be surprised if they are doing especially poorly in the predict_proba. class_weight='auto' may help some with this. Jul 2, 2015 at 4:58

1 Answer 1

1

I don't know much about how SVC works, so you may consider what is said in the comment to complete this answer.

You have to consider that predic_proba will give you the categories in a lexicographical order as they appear in the classes_ attribute. You have this in the doc.

When you want to print your result you have to consider this. And we can see on your examples that Angry is at the first index so your result are good except for the first one.

try this :

print 'Neutral: ', prob_s[0,1]
print 'Smiling: ', prob_s[0,3]
print 'Shocked: ', prob_s[0,2]
print 'Angry: ', prob_s[0,0]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.