I'm coding in python.
In my work, I have a certain parameter for which there are multiple possible values, as well some data X
with associated binary target values y
.
For each of the parameter's values, I run some classification algorithm (eg.: Naive Bayes) using scickit-learn on X
and y
.
In this step, we use cross validation, doing an 80:20 train-test split on our dataset across some number of folds, and average the results.
This yields a score (the area under the precision recall curve) for each parameter value, and we pick the one with highest score to be our preferred parameter value for this classifier.
We would then like to train the algorithm using all of the data X
, rather than just 80% of it, obtaining some decision function f
which we'll use for making predictions.
These decision functions return probability or numerical values that indicate 'how sure' the algorithm is a certain sample belongs to a certain class; they don't properly predict a class for a sample.
This is done with threshold values t
: samples for which the numerical value is less than t
are assigned the class 0, and the rest of the sampels are assigned the class 1.
When we have the true labels, these predictions can be tested against the true labels to make evaluations (like precision and recall) about the prediction.
Varying the value of t
is precisely what generates the multiple points in Precision-Recall space, allowing us to plot a curve (and hence calculate the area under it) for a given model.
We did this for each fold in our cross validation.
Now, we have some data Z
to which we can apply f
, but we don't have the labels.
How can we choose an appropriate value of t
for our model?
More specifically, consider the models generated during the cross-validation (for the same preferred parameter value). Each such model has a corresponding decision function, and hence a corresponding precision-recall curve with associated threshold values.
Given a point p_0
in one of these curves, and hence an associated threshold t_0
value, is it reasonable to assume that points close to p_0
in other curves will have associated threshold values close to t_0
?
In other words, is it reasonable to expect our model (trained with all of the data) to behave similarly to those we got in cross validation if we use as threshold values the average of the threshold values obtained during cross validation?
You can assume there is little variance between the average of the models obtained in cross validation and each cross validation model itself. For the previously mentioned Naive Bayes classifier, we for instance have
so different folds produce curves that stick very close together.