7

I am training CIFAR10 dataset on LeNet CNN model. I am using PyTorch on Google Colab. The code runs only when I use Adam optimizer with model.parameters() as the only parameter. But when I change my optimizer or use weight_decay parameter then the accuracy remains at 10% through all the epochs. I cannot understand the reason why it is happening.

# CNN Model - LeNet    
class LeNet_ReLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn_model = nn.Sequential(nn.Conv2d(3,6,5), 
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2), 
                                       nn.Conv2d(6,16,5), 
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2))  
        self.fc_model = nn.Sequential(nn.Linear(400, 120),   
                                      nn.ReLU(),
                                      nn.Linear(120,84),  
                                      nn.ReLU(),
                                      nn.Linear(84,10))

    def forward(self, x):
        x = self.cnn_model(x)
        x = x.view(x.size(0), -1)
        x = self.fc_model(x)
        return x

# Importing dataset and creating dataloader
batch_size = 128
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True,
                                    transform=transforms.ToTensor())
trainloader = utils_data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True,
                                    transform=transforms.ToTensor())
testloader = utils_data.DataLoader(testset, batch_size=batch_size, shuffle=False)

# Creating instance of the model
net = LeNet_ReLU()

# Evaluation function
def evaluation(dataloader):
    total, correct = 0, 0
    for data in dataloader:
        inputs, labels = data

        outputs = net(inputs)
        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (pred==labels).sum().item()
    return correct/total * 100

# Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = optim.Adam(net.parameters(), weight_decay = 0.9)

# Model training
loss_epoch_arr = []
max_epochs = 16

for epoch in range(max_epochs):
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data

        outputs = net(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        opt.step()

        opt.zero_grad()


    loss_epoch_arr.append(loss.item())

    print('Epoch: %d/%d, Test acc: %0.2f, Train acc: %0.2f'
    % (epoch,max_epochs, evaluation(testloader), evaluation(trainloader))) 

plt.plot(loss_epoch_arr)
5
  • Did you trained it without the weight decay configuration? How did it performed on the test data? in terms of accuracy
    – A. Maman
    Commented May 6, 2020 at 13:10
  • 1
    I trained with the weight_decay configuration. The accuracy stayed at 10% for both train and test data for all the epochs. It didn't change. But when I tried without it, the train and test accuracy increases to 66% and 56% respectively.. Commented May 6, 2020 at 16:09
  • 5
    weight_decay=0.9 is wayyyy too high. Basically this is instructing the optimizer that having small weights is much more important than having a low loss value. A common value is weight_decay=0.0005 or within an order of magnitude of that.
    – jodag
    Commented May 6, 2020 at 17:23
  • It is working now. I was adding too big a value of weight_decay. Thanks. Commented May 7, 2020 at 18:12
  • When I asked chatgpt, it explained that a weight decay of .9 means that it reduces the value of each parameter by 10% each iteration! That's a big amount! Commented Apr 18 at 2:08

1 Answer 1

7

The weight decay mechanism sets a penalty for high value weights, i.e. it stricts the weights to have relatively small values by adding their sum multiplied by the weight_decay argument you gave it. That can be seen as a quadratic regularization term.

When passing large weight_decay value, you may strict your network too much and prevent it from learning, that's probably the reason it had 10% of accuracy which is related to non-learning at all and just guessing the answer (since you have 10 classes you receive 10% of acc, when the output isn't a function of your input at all).

The solution would be to play around with different values, train for weight_decay of 1e-4 or some other values in that area. Note that when you reach values closer to zero you should have results which are closer to your initial train without using the weight decay.

Hope that helps.

2
  • It helped. Thanks a lot. Commented May 7, 2020 at 18:12
  • Great! if it did aswered your questions, please mark it as "aswered" please (the V sign below the voting arrows). Thanks!
    – A. Maman
    Commented May 8, 2020 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.