10

I'm doing fine-tuning with pytorch using resnet50 and want to set the learning rate of the last fully connected layer to 10^-3 while the learning rate of other layers be set to 10^-6. I know that I can just follow the method in its document:

optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}], 
          lr=1e-2, momentum=0.9)

But is there anyway that I do not need to set the parameters layer by layer

1
  • 2
    Can you clarify your question.. You would want to set the learning rate of the last fully connected layer to 1e-3 and the rest to 1e-6?? What do you mean by "But is there anyway that I do not need to set the parameters layer by layer?"
    – Kashyap
    Commented May 13, 2017 at 4:39

1 Answer 1

1

You can group layers. If you want to group all linear layers, the best way to do it is use modules:

param_grp = []

for idx, m in enumerate(model.modules()):
    if isinstance(m, nn.Linear):
        param_grp.append(m.weight)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.