5

What's the overall point of training error in the goal of regression (i.e, making predictions)?

You might say something like, "well, you see, training error can help you determine which model of complexity is the best to use. "

And to that, some would say, "No you can't. Low training error could just mean that your model is conforming to whatever data you're training the model with, A.K.A overfitting"

What's the point of calculating training error if it's not a good predictive measure of performance?

Especially when we go through and say, to hell with training error, just use validation error..

When will we ever use training error?

Low training error can be indicative of overfitting.. is that the only use of it?

2
  • 1
    Stack Overflow is probably not the best place for this question. Take a look at this Meta Stack Exchange Question for some more appropriate suggestions.
    – skrrgwasme
    Aug 8, 2016 at 16:59
  • that's a good suggestion
    – user2738183
    Aug 8, 2016 at 19:55

1 Answer 1

7

Training error by itself can be a very bad metric of your model performance, as you have correctly pointed out. However, there is no going around the fact that you need to train your model to make some meaningful predictions.

That is why you need training, validation and the test phases and data sets. The overfitting that can easily happen in the training dataset can be alleviated to some extent by using the randomly sub-sampled validation dataset because if you have overfitted, your model will not generalize (you should see that your training error does down monotonically as the model complexity increases but your validation error plateaus at some point and additional model complexity actually increases the validation error). However, if you do not do any training of the model, you do not have a model to validate!

The model needs to be trained. There is no getting around that. However, training error by itself is useless. One needs to perform cross-validation in order to ensure that the model is generalizable. The bottom line is that anything you use which the model has seen to during the training phase to evaluate it's performance is invalid. It is useful for model fitting but not for evaluation. The correct way to do so is cross-validation regardless of what the OP claims in the discussion below.

You should look into the concept of bias-variance tradeoff as this has a direct bearing on your question and should clarify your doubt.

21
  • So basically you're saying that training error is useless except to help spot overfitting. Is that correct?
    – user2738183
    Aug 8, 2016 at 14:53
  • By itself, it has little value, yes. It cannot spot overfitting. That bit is done by the validation dataset. The training phase can overfit especially for complex models and that is when cross validation is very critical. You are over thinking it...training error is just the error between the true labels and the predicted labels during the training phase. It is just the residue left over after the optimization. There is nothing special about the training error per se. It is just that you need to cross-validate after the training phase to ensure that your model is generalizable. Aug 8, 2016 at 14:56
  • 1
    No no no.. low training error can HELP spot overfitting. 'help' was a little word you missed there ;). Why does it help? It helps because if you have low training error but high test error, you're overfitting. The point of the validation set is so that you can choose the correct tuning parameters.. such as lamda star in ridge regression. I have to split this up into several comments...
    – user2738183
    Aug 8, 2016 at 15:02
  • You can spot overfitting in other ways, such as coefficients that are really large in magnitude which ridge regression helps take care of.
    – user2738183
    Aug 8, 2016 at 15:03
  • 1
    And actually, training error is usually defined as the average of rss, or whatever loss function you would like to use. Rss being sum((actual -prediction)^2) maybe I'm being too technical with the details..
    – user2738183
    Aug 8, 2016 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.