3

The title says it all: Should a neural network be able to have a perfect train accuracy? Mine saturates at ~0.9 accuracy and I am wondering if that indicates a problem with my network or the training data.

Training instances: ~4500 sequences with an average length of 10 elements. Network: Bi-directional vanilla RNN with a softmax layer on top.

2 Answers 2

5

Perfect accuracy on training data is usually a sign of a phenomenon called overfitting (https://en.wikipedia.org/wiki/Overfitting) and the model may generalize poorly to unseen data. So, no, probably this alone is not an indication that there is something wrong (you could still be overfitting but it is not possible to tell from the information in your question).

You should check the accuracy of the NN on the validation set (data your network has not seen during training) and judge its generalizability. usually it's an iterative process where you train many networks with different configurations in parallel and see which one performs best on the validation set. Also see cross validation (https://en.wikipedia.org/wiki/Cross-validation_(statistics))

If you have low measurement noise, a model may still not get zero training error. This could be for many reasons including that the model is not flexible enough to capture the true underlying function (which can be a complicated, high-dimensional, non-linear function). You can try increasing the number of hidden layers and nodes but you have to be careful about the same things like overfitting and only judge based on evaluation through cross validation.

You can definitely get a 100% accuracy on training datasets by increasing model complexity but I would be wary of that.

2
  • Thank you for your answer. I am aware of overfitting, I really mean only accuracy on the training set. If the training data is not noisy, and if the network works properly, should it reach perfect accuracy on the training set? Or are there cases where a network never can reach perfect accuracy on the training data?
    – Alex
    Aug 18, 2016 at 9:32
  • 1
    If you have low measurement noise, a model may still not get zero training error. This could be for many reasons including that the model is not flexible enough to capture the true underlying function (which can be a very complicated non-linear function). You can try increasing the number of hidden layers and nodes but you have to be careful about the same things like overfitting and only judge based on evaluation through cross validation. You can definitely get a 100% accuracy on training datasets by increasing model complexity but I would be wary of that. Aug 18, 2016 at 9:41
2

You cannot expect your model to be better on your test set than on your training set. This means if your training accuracy is lower than the desired accuracy, you have to change something. Most likely you have to increase the number of parameters of your model.

The reason why you might be ok with not having a perfect training accuracy is (1) the problem of overfitting (2) training time. The more complex your model is, the more likely is overfitting.

You might want to have a look at Structural Risc Minimization:


(source: svms.org)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.