1

I'm new in ML. While I was reading about backpropagation Suddenly, I have question.
In backpropagation learning of a neural network,
Do we should start with a small learning rate and slowly increase it during the learning process? or
Do we should start with big learning rate and slowly reduce it during the learning process?

Which one is correct?

1

2 Answers 2

1

Generally, the second one it correct - Think about this in this way - big learning rate means that you roughly search for the best area in the space. Then, with a small learning you tune the weights to find the best value.

If you would use constant big learning rate you would "jump" around the minimum point. If you would use constant small learning rate it would take a lot of time to converge. That`s why learning rate decaying is a good idea.

enter image description here

Having said that, there are a few more advanced tricks for learning rate scheduling, that are not monotonically decreasing the learning rate.

0

When finetuning the learning rate, it's important to look whether your model tends to overfit or underfit.

A good indication for overfitting is when your model performs incredibly well on training data, but poorly on test data. Like in the left picture above, you can see how the model adjusts to every point during training, without learning the underlying pattern. Underfitting is straight forward: Your model doesn't perform. Neither on training, nor on test data.

As a general rule, when your model overfits, the learning rate might be too high. When your model underfits, the learning rate might be too low.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.