93

Is it possible in PyTorch to change the learning rate of the optimizer in the middle of training dynamically (I don't want to define a learning rate schedule beforehand)?

So let's say I have an optimizer:

optim = torch.optim.SGD(model.parameters(), lr=0.01)

Now due to some tests which I perform during training, I realize my learning rate is too high so I want to change it to say 0.001. There doesn't seem to be a method optim.set_lr(0.001) but is there some way to do this?

2 Answers 2

182

So the learning rate is stored in optim.param_groups[i]['lr']. optim.param_groups is a list of the different weight groups which can have different learning rates. Thus, simply doing:

for g in optim.param_groups:
    g['lr'] = 0.001

will do the trick.


**Alternatively**,

as mentionned in the comments, if your learning rate only depends on the epoch number, you can use a learning rate scheduler.

For example (modified example from the doc):

from torch.optim.lr_scheduler import LambdaLR
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
# Assuming the optimizer has two groups.
lambda_group1 = lambda epoch: epoch // 30
lambda_group2 = lambda epoch: 0.95 ** epoch
scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

Also, there is a prebuilt learning rate scheduler to reduce on plateaus.

4
  • @MehmetBurakSayıcı ask a new question. Commented Aug 20, 2019 at 20:01
  • 3
    Use this automatic updaters in 2020
    – A.Ametov
    Commented May 19, 2020 at 17:54
  • 2
    i tried to use this but i didn't change the lr. had to make the change like this: for i in range(len(optimizer.param_groups)): optimizer.param_groups[i]['lr'] = new_lr Commented Apr 11, 2021 at 18:35
  • 1
    Note that if you use both techniques together, that is you set the learning rate manually and you then use a scheduler, you should not set the lr field of the parameter group but the initial_lr field (or both). The scheduler uses this to set the lr field from, so it'll override any manual changes you make.
    – Peter
    Commented Jan 1 at 11:49
21

Instead of a loop in patapouf_ai's answer, you can do it directly via:

optim.param_groups[0]['lr'] = 0.001
3
  • 15
    This only works if you have a single parameter group. (Which granted is probably most of the time.) Commented Nov 4, 2020 at 19:43
  • for some reason this makes model not learn at all Commented Nov 21, 2023 at 17:33
  • 1
    yeah well cause I did it per batch instead of per epoch, this is rather a confirmation that it works Commented Nov 21, 2023 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.