7

Is there any built-in function to get the maximum accuracy for a binary probabilistic classifier in scikit-learn?

E.g. to get the maximum F1-score I do:

# AUCPR
precision, recall, thresholds = sklearn.metrics.precision_recall_curve(y_true, y_score)    
auprc  = sklearn.metrics.auc(recall, precision)
max_f1 = 0
for r, p, t in zip(recall, precision, thresholds):
    if p + r == 0: continue
    if (2*p*r)/(p + r) > max_f1:
        max_f1 = (2*p*r)/(p + r) 
        max_f1_threshold = t

I could compute the maximum accuracy in a similar fashion:

accuracies = []
thresholds = np.arange(0,1,0.1)
for threshold in thresholds:
    y_pred = np.greater(y_score, threshold).astype(int)
    accuracy = sklearn.metrics.accuracy_score(y_true, y_pred)
    accuracies.append(accuracy)

accuracies = np.array(accuracies)
max_accuracy = accuracies.max() 
max_accuracy_threshold =  thresholds[accuracies.argmax()]

but I wonder whether there is any built-in function.

5
  • Hi Franck, did you find a built-in function for it, because I'm searching the same right now.
    – Geeocode
    Commented Jan 8, 2016 at 11:37
  • 1
    @GeorgeSolymosi I didnt find a built-in function for it. Commented Jan 8, 2016 at 15:36
  • 1
    Thanks for the info, note, that the row accuracy = np.array(accuracy) should have been altered to accuracy = np.array(accuracies) or similar:)
    – Geeocode
    Commented Jan 8, 2016 at 15:44
  • @GeorgeSolymosi Thanks good catch! Commented Jan 8, 2016 at 15:46
  • yw, by the way nice, clear, transparent code Franck!
    – Geeocode
    Commented Jan 8, 2016 at 16:02

2 Answers 2

6
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_true, probs)
accuracy_scores = []
for thresh in thresholds:
    accuracy_scores.append(accuracy_score(y_true, [m > thresh for m in probs]))

accuracies = np.array(accuracy_scores)
max_accuracy = accuracies.max() 
max_accuracy_threshold =  thresholds[accuracies.argmax()]

2

I started to improve the solution by transforming the thresholds = np.arange(0,1,0.1) into a smarter, dichotomous way of finding the maximum

Then I realized, after 2 hours of work, that getting all the accuracies were far more cheaper than just finding the maximum !! (Yes it is totally counter-intuitive).

I wrote a lot of comments here below to explain my code. Feel free to delete all these to make the code more readable.

import numpy as np

# Definition : we predict True if y_score > threshold
def ROC_curve_data(y_true, y_score):
    y_true  = np.asarray(y_true,  dtype=np.bool_)
    y_score = np.asarray(y_score, dtype=np.float_)
    assert(y_score.size == y_true.size)

    order = np.argsort(y_score) # Just ordering stuffs
    y_true  = y_true[order]
    # The thresholds to consider are just the values of score, and 0 (accept everything)
    thresholds = np.insert(y_score[order],0,0)
    TP = [sum(y_true)] # Number of True Positives (For Threshold = 0 => We accept everything => TP[0] = # of postive in true y)
    FP = [sum(~y_true)] # Number of True Positives (For Threshold = 0 => We accept everything => TP[0] = # of postive in true y)
    TN = [0] # Number of True Negatives (For Threshold = 0 => We accept everything => we don't have negatives !)
    FN = [0] # Number of True Negatives (For Threshold = 0 => We accept everything => we don't have negatives !)

    for i in range(1, thresholds.size) : # "-1" because the last threshold
        # At this step, we stop predicting y_score[i-1] as True, but as False.... what y_true value say about it ?
        # if y_true was True, that step was a mistake !
        TP.append(TP[-1] - int(y_true[i-1]))
        FN.append(FN[-1] + int(y_true[i-1]))
        # if y_true was False, that step was good !
        FP.append(FP[-1] - int(~y_true[i-1]))
        TN.append(TN[-1] + int(~y_true[i-1]))

    TP = np.asarray(TP, dtype=np.int_)
    FP = np.asarray(FP, dtype=np.int_)
    TN = np.asarray(TN, dtype=np.int_)
    FN = np.asarray(FN, dtype=np.int_)

    accuracy    = (TP + TN) / (TP + FP + TN + FN)
    sensitivity = TP / (TP + FN)
    specificity = TN / (FP + TN)
    return((thresholds, TP, FP, TN, FN))

The all process is just a single loop, and the algorithm is just trivial. In fact, the stupidly simple function is 10 times faster than the solution proposed before me (commpute the accuracies for thresholds = np.arange(0,1,0.1)) and 30 times faster than my previous smart-ass-dychotomous-algorithm...

You can then easily compute ANY KPI you want, for example :

def max_accuracy(thresholds, TP, FP, TN, FN) :
    accuracy    = (TP + TN) / (TP + FP + TN + FN)
    return(max(accuracy))

def max_min_sensitivity_specificity(thresholds, TP, FP, TN, FN) :
    sensitivity = TP / (TP + FN)
    specificity = TN / (FP + TN)
    return(max(np.minimum(sensitivity, specificity)))

If you want to test it :

y_score = np.random.uniform(size = 100)
y_true = [np.random.binomial(1, p) for p in y_score]
data = ROC_curve_data(y_true, y_score)

%matplotlib inline # Because I personnaly use Jupyter, you can remove it otherwise
import matplotlib.pyplot as plt
plt.step(data[0], data[1])
plt.step(data[0], data[2])
plt.step(data[0], data[3])
plt.step(data[0], data[4])
plt.show()

print("Max accuracy is", max_accuracy(*data))
print("Max of Min(Sensitivity, Specificity) is", max_min_sensitivity_specificity(*data))

Enjoy ;)

1
  • 2
    The downside of this is that, particularly for imbalanced datasets, most of the variation in score may lie in the first or last bin. A better method computes threshold,tp,fp,fn,tn for each unique (tp,fp,fn,tn). This can be done efficiently in a single pass (scikit does this internally when computing AUCROC.)
    – user48956
    Commented Jul 17, 2017 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.