8

I'm using cross_val_score from scikit-learn (package sklearn.cross_validation) to evaluate my classifiers.
If I use f1 for the scoring parameter, the function will return the f1-score for one class. To get the average I can use f1_weighted but I can't find out how to get the f1-score of the other class. (precision and recall analogous)

The functions in sklearn.metrics have a labels parameter which does this, but I can't find anything like this in the documentation.

Is there a way to get the f1-score for all classes at once or at least specify the class which should be considered with cross_val_score?

4 Answers 4

9

When you create a scorer with make_scorer function you can pass any additional arguments you need, like this:

cross_val_score(
    svm.SVC(kernel='rbf', gamma=0.7, C = 1.0),
    X, y,
    scoring=make_scorer(f1_score, average='weighted', labels=[2]),
    cv=10)

But cross_val_score only allows you to return one score. You can't get scores for all classes at once without additional tricks. If you need that please refer to another stack overflow question which covers exactly that: Evaluate multiple scores on sklearn cross_val_score

0

You may simply try the following:

svm = LinearSVC()
scores = cross_val_score(svm, X, y, 
                         scoring = "f1",
                         cv = 10)
-1

For individual scores of each class, use this :

f1 = f1_score(y_test, y_pred, average= None) print("f1 list non intent: ", f1)

-4

To compute F1 score, we can use sklearn.metrics.f1_score

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html

Sample code

from sklearn import svm
from sklearn import metrics
from sklearn.cross_validation import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import f1_score, accuracy_score

# prepare dataset
iris = load_iris()
X = iris.data[:, :2]
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# svm classification
clf = svm.SVC(kernel='rbf', gamma=0.7, C = 1.0).fit(X_train, y_train)
y_predicted = clf.predict(X_test)

# performance
print "Classification report for %s" % clf
print metrics.classification_report(y_test, y_predicted)

print("F1 micro: %1.4f\n" % f1_score(y_test, y_predicted, average='micro'))
print("F1 macro: %1.4f\n" % f1_score(y_test, y_predicted, average='macro'))
print("F1 weighted: %1.4f\n" % f1_score(y_test, y_predicted, average='weighted'))
print("Accuracy: %1.4f" % (accuracy_score(y_test, y_predicted)))

Sample Output

Classification report for SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape=None, degree=3, gamma=0.7, kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)
             precision    recall  f1-score   support

          0       1.00      0.90      0.95        10
          1       0.50      0.88      0.64         8
          2       0.86      0.50      0.63        12

avg / total       0.81      0.73      0.74        30

F1 micro: 0.7333

F1 macro: 0.7384

F1 weighted: 0.7381

Accuracy: 0.7333
1
  • 2
    Thanks, but that is not what I asked for. cross_val_score does the cross-validation and calculates the scores. With your method I would still have to implement the cross-validation and aggregation of the scores, plus I still don't get the f1 score per class. I have a working implementation but I'd like to use as many standard functions to simplify the code.
    – toydarian
    Commented May 25, 2016 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.