6

Using the below precision recall graph where recall is on x-axis and precision is on y-axis can I use this formula to calculate the number of predictions for a given precision, recall threshold ?

enter image description here

These calculations are based on orange trend line.

Assuming this model has been trained on 100 instances and is a binary classifier.

At recall value 0.2 there (0.2 * 100) = 20 relevant instances. At recall value 0.2 the precision = .95 so the number of true positives (20 * .95) = 19. Is this a correct method to calculate the number of true positives from precision-recall graph ?

1

5 Answers 5

4
+100

I will argue it is not possible to do it like this. For the ease of calculations I will take a recall of 20%, precision of 90% and 100 observations.

I can make two result matrices which will produce those numbers. Here TP/TN denote Test Positive and Negative, CP/CN denote Condition Positive/Negative:

   CP CN
TP 9 1
TN 36 54

and

   CP CN
TP 18 2
TN 72 8

Matrix 1 has TP of 9, FP of 1 and FN of 36, resulting in a recall of 9 / (36 + 9) = 20% and a precision of 9 / (1 + 9) = 90%

Matrix 2 has TP of 18, FP of 2 and FN of 72, resulting in a recall of 18 / (72 + 18) = 20% and a precision of 18 / (2+18) = 90%

Since I can produce two matrices with different TP and the same recall + precision, the graph does not give enough information to trace back the TP.

2
  • It is explicitly clarified that we are talking about a binary classifier. I see your point though...
    – desertnaut
    Mar 21, 2019 at 14:51
  • You're right, dunno how I missed that. Updated the answer.
    – CIAndrews
    Mar 21, 2019 at 23:33
0

I am not sure of what exactly you mean, but, I would think of this as follows:

Recall = TP/ (TP + FN) which in your case you got it correct = 20 relevant instances among all the instances that were classified are positive.

Precision = TP/ (TP + FP) which in your case you say is 0.95 means 95 instances out of 100 were classified correctly at that point.

Now lets equate the two for our case:

0.2 = TP/ (TP + FN) and 0.95 = TP/ (TP + FP)

Therefore,

0.75 TP = 0.2*FN - 0.95*FP

-> TP = (0.2*FN - 0.95*FP)/ 0.75

I would calculate the actual True Positives in my data from the above equation.

When you multiply predicted relevant samples with precision, you are just calculating the instances that were predicted TP and were relevant. I am not sure if it accounts for all the True Positives in your data.

However, you can say with good certainty (basically you are correct) that your model predicted them as relevant TPs, if that's what you are looking for..

Hope this helps!

3
  • 1
    To calculate TP from the subject equation presupposes of course that you already know FP and FN...
    – desertnaut
    Mar 18, 2019 at 16:33
  • @desertnaut Then I guess my answer seems okay, right? Mar 18, 2019 at 18:44
  • Does it? OP asks how to calculate TPs from the curve, so if you can't provide a way to calculate FNs & FPs this certainly cannot be done from the equation...
    – desertnaut
    Mar 18, 2019 at 18:46
0

No,

E.g :- Recall = 0.2, Precision = 0.95, 100 data points

say, tp = True+ve, fp = False+ve , fn = False-ve, tn = True-ve

with your current method.

tp = Precision * total number of data points

or

Precision = tp / (total number of data points)

Actual definition precision states

Precision = tp / (tp+fp)

For your calculation to work, below condition should be true

 tp + fp = total number of data points

but

total number of data points = tp + fp + tn + fn
-1

Use python.If you need any more modification , question , Have a look here Collected from : https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html

"""
================
Precision-Recall
================

Example of Precision-Recall metric to evaluate classifier output quality.

Precision-Recall is a useful measure of success of prediction when the
classes are very imbalanced. In information retrieval, precision is a
measure of result relevancy, while recall is a measure of how many truly
relevant results are returned.

The precision-recall curve shows the tradeoff between precision and
recall for different threshold. A high area under the curve represents
both high recall and high precision, where high precision relates to a
low false positive rate, and high recall relates to a low false negative
rate. High scores for both show that the classifier is returning accurate
results (high precision), as well as returning a majority of all positive
results (high recall).

A system with high recall but low precision returns many results, but most of
its predicted labels are incorrect when compared to the training labels. A
system with high precision but low recall is just the opposite, returning very
few results, but most of its predicted labels are correct when compared to the
training labels. An ideal system with high precision and high recall will
return many results, with all results labeled correctly.

Precision (:math:`P`) is defined as the number of true positives (:math:`T_p`)
over the number of true positives plus the number of false positives
(:math:`F_p`).

:math:`P = \\frac{T_p}{T_p+F_p}`

Recall (:math:`R`) is defined as the number of true positives (:math:`T_p`)
over the number of true positives plus the number of false negatives
(:math:`F_n`).

:math:`R = \\frac{T_p}{T_p + F_n}`

These quantities are also related to the (:math:`F_1`) score, which is defined
as the harmonic mean of precision and recall.

:math:`F1 = 2\\frac{P \\times R}{P+R}`

Note that the precision may not decrease with recall. The
definition of precision (:math:`\\frac{T_p}{T_p + F_p}`) shows that lowering
the threshold of a classifier may increase the denominator, by increasing the
number of results returned. If the threshold was previously set too high, the
new results may all be true positives, which will increase precision. If the
previous threshold was about right or too low, further lowering the threshold
will introduce false positives, decreasing precision.

Recall is defined as :math:`\\frac{T_p}{T_p+F_n}`, where :math:`T_p+F_n` does
not depend on the classifier threshold. This means that lowering the classifier
threshold may increase recall, by increasing the number of true positive
results. It is also possible that lowering the threshold may leave recall
unchanged, while the precision fluctuates.

The relationship between recall and precision can be observed in the
stairstep area of the plot - at the edges of these steps a small change
in the threshold considerably reduces precision, with only a minor gain in
recall.

**Average precision** (AP) summarizes such a plot as the weighted mean of
precisions achieved at each threshold, with the increase in recall from the
previous threshold used as the weight:

:math:`\\text{AP} = \\sum_n (R_n - R_{n-1}) P_n`

where :math:`P_n` and :math:`R_n` are the precision and recall at the
nth threshold. A pair :math:`(R_k, P_k)` is referred to as an
*operating point*.

AP and the trapezoidal area under the operating points
(:func:`sklearn.metrics.auc`) are common ways to summarize a precision-recall
curve that lead to different results. Read more in the
:ref:`User Guide <precision_recall_f_measure_metrics>`.

Precision-recall curves are typically used in binary classification to study
the output of a classifier. In order to extend the precision-recall curve and
average precision to multi-class or multi-label classification, it is necessary
to binarize the output. One curve can be drawn per label, but one can also draw
a precision-recall curve by considering each element of the label indicator
matrix as a binary prediction (micro-averaging).

.. note::

    See also :func:`sklearn.metrics.average_precision_score`,
             :func:`sklearn.metrics.recall_score`,
             :func:`sklearn.metrics.precision_score`,
             :func:`sklearn.metrics.f1_score`
"""
from __future__ import print_function

###############################################################################
# In binary classification settings
# --------------------------------------------------------
#
# Create simple data
# ..................
#
# Try to differentiate the two first classes of the iris data
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
import numpy as np

iris = datasets.load_iris()
X = iris.data
y = iris.target

# Add noisy features
random_state = np.random.RandomState(0)
n_samples, n_features = X.shape
X = np.c_[X, random_state.randn(n_samples, 200 * n_features)]

# Limit to the two first classes, and split into training and test
X_train, X_test, y_train, y_test = train_test_split(X[y < 2], y[y < 2],
                                                    test_size=.5,
                                                    random_state=random_state)

# Create a simple classifier
classifier = svm.LinearSVC(random_state=random_state)
classifier.fit(X_train, y_train)
y_score = classifier.decision_function(X_test)

###############################################################################
# Compute the average precision score
# ...................................
from sklearn.metrics import average_precision_score
average_precision = average_precision_score(y_test, y_score)

print('Average precision-recall score: {0:0.2f}'.format(
      average_precision))

###############################################################################
# Plot the Precision-Recall curve
# ................................
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt
from sklearn.utils.fixes import signature

precision, recall, _ = precision_recall_curve(y_test, y_score)

# In matplotlib < 1.5, plt.fill_between does not have a 'step' argument
step_kwargs = ({'step': 'post'}
               if 'step' in signature(plt.fill_between).parameters
               else {})
plt.step(recall, precision, color='b', alpha=0.2,
         where='post')
plt.fill_between(recall, precision, alpha=0.2, color='b', **step_kwargs)

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(
          average_precision))

###############################################################################
# In multi-label settings
# ------------------------
#
# Create multi-label data, fit, and predict
# ...........................................
#
# We create a multi-label dataset, to illustrate the precision-recall in
# multi-label settings

from sklearn.preprocessing import label_binarize

# Use label_binarize to be multi-label like settings
Y = label_binarize(y, classes=[0, 1, 2])
n_classes = Y.shape[1]

# Split into training and test
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=.5,
                                                    random_state=random_state)

# We use OneVsRestClassifier for multi-label prediction
from sklearn.multiclass import OneVsRestClassifier

# Run classifier
classifier = OneVsRestClassifier(svm.LinearSVC(random_state=random_state))
classifier.fit(X_train, Y_train)
y_score = classifier.decision_function(X_test)


###############################################################################
# The average precision score in multi-label settings
# ....................................................
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score

# For each class
precision = dict()
recall = dict()
average_precision = dict()
for i in range(n_classes):
    precision[i], recall[i], _ = precision_recall_curve(Y_test[:, i],
                                                        y_score[:, i])
    average_precision[i] = average_precision_score(Y_test[:, i], y_score[:, i])

# A "micro-average": quantifying score on all classes jointly
precision["micro"], recall["micro"], _ = precision_recall_curve(Y_test.ravel(),
    y_score.ravel())
average_precision["micro"] = average_precision_score(Y_test, y_score,
                                                     average="micro")
print('Average precision score, micro-averaged over all classes: {0:0.2f}'
      .format(average_precision["micro"]))

###############################################################################
# Plot the micro-averaged Precision-Recall curve
# ...............................................
#

plt.figure()
plt.step(recall['micro'], precision['micro'], color='b', alpha=0.2,
         where='post')
plt.fill_between(recall["micro"], precision["micro"], alpha=0.2, color='b',
                 **step_kwargs)

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title(
    'Average precision score, micro-averaged over all classes: AP={0:0.2f}'
    .format(average_precision["micro"]))

###############################################################################
# Plot Precision-Recall curve for each class and iso-f1 curves
# .............................................................
#
from itertools import cycle
# setup plot details
colors = cycle(['navy', 'turquoise', 'darkorange', 'cornflowerblue', 'teal'])

plt.figure(figsize=(7, 8))
f_scores = np.linspace(0.2, 0.8, num=4)
lines = []
labels = []
for f_score in f_scores:
    x = np.linspace(0.01, 1)
    y = f_score * x / (2 * x - f_score)
    l, = plt.plot(x[y >= 0], y[y >= 0], color='gray', alpha=0.2)
    plt.annotate('f1={0:0.1f}'.format(f_score), xy=(0.9, y[45] + 0.02))

lines.append(l)
labels.append('iso-f1 curves')
l, = plt.plot(recall["micro"], precision["micro"], color='gold', lw=2)
lines.append(l)
labels.append('micro-average Precision-recall (area = {0:0.2f})'
              ''.format(average_precision["micro"]))

for i, color in zip(range(n_classes), colors):
    l, = plt.plot(recall[i], precision[i], color=color, lw=2)
    lines.append(l)
    labels.append('Precision-recall for class {0} (area = {1:0.2f})'
                  ''.format(i, average_precision[i]))

fig = plt.gcf()
fig.subplots_adjust(bottom=0.25)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Extension of Precision-Recall curve to multi-class')
plt.legend(lines, labels, loc=(0, -.38), prop=dict(size=14))


plt.show()
-2

Since you are able to plot a precision-recall curve, i am going to take that you have the precision and recall values in some variable.

Let's assume precision=0.75

0.75 can be written as 3/4

fraction=(0.75).as_integer_ratio()

output:

(3, 4)

If the number of items you have is 100,

numerator=3*100/(3+4)

nr=(fraction[0]*100)/sum(fraction)

denominator=4*100/(3+4)

dr=(fraction[1]*100)/sum(fraction)

The formula for precision is TP/(TP+FP)

therefore TP=numerator and FP=denominator-TP

tp=nr
fp=dr-tp

Similarly, we can calculate FN from recall

Your result might be a decimal value and since TP,TN,FP,FN, cant be a fraction, we can round the value to the nearest 1.

I hope this is helpful!

Not the answer you're looking for? Browse other questions tagged or ask your own question.