0

I'm coding in python. In my work, I have a certain parameter for which there are multiple possible values, as well some data X with associated binary target values y.

For each of the parameter's values, I run some classification algorithm (eg.: Naive Bayes) using scickit-learn on X and y. In this step, we use cross validation, doing an 80:20 train-test split on our dataset across some number of folds, and average the results.

This yields a score (the area under the precision recall curve) for each parameter value, and we pick the one with highest score to be our preferred parameter value for this classifier. We would then like to train the algorithm using all of the data X, rather than just 80% of it, obtaining some decision function f which we'll use for making predictions.


These decision functions return probability or numerical values that indicate 'how sure' the algorithm is a certain sample belongs to a certain class; they don't properly predict a class for a sample. This is done with threshold values t: samples for which the numerical value is less than t are assigned the class 0, and the rest of the sampels are assigned the class 1.

When we have the true labels, these predictions can be tested against the true labels to make evaluations (like precision and recall) about the prediction. Varying the value of t is precisely what generates the multiple points in Precision-Recall space, allowing us to plot a curve (and hence calculate the area under it) for a given model. We did this for each fold in our cross validation.


Now, we have some data Z to which we can apply f, but we don't have the labels. How can we choose an appropriate value of t for our model?

More specifically, consider the models generated during the cross-validation (for the same preferred parameter value). Each such model has a corresponding decision function, and hence a corresponding precision-recall curve with associated threshold values.

Given a point p_0 in one of these curves, and hence an associated threshold t_0 value, is it reasonable to assume that points close to p_0 in other curves will have associated threshold values close to t_0? In other words, is it reasonable to expect our model (trained with all of the data) to behave similarly to those we got in cross validation if we use as threshold values the average of the threshold values obtained during cross validation?


You can assume there is little variance between the average of the models obtained in cross validation and each cross validation model itself. For the previously mentioned Naive Bayes classifier, we for instance have

Example plot with multiple precision-recall curves: one for each cross validation fold, as well as the mean curve

so different folds produce curves that stick very close together.

3
  • 1
    I'm voting to close this question as off-topic because this question belongs on Data Science.
    – pault
    Apr 13, 2018 at 18:03
  • If your ROC curves are drawn like on the wiki page, one way is to pick the threshold that corresponds to the point closest to (0, 1). Another option is to pick the point that maximizes the KS stat (biggest difference between the curve and the line y=x). It really depends on your end goal and you'll get better answers on Data Science.
    – pault
    Apr 13, 2018 at 18:06
  • I will cross post this to Data Science, thanks. Apr 13, 2018 at 19:06

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy