I have a question about the following code in pytorch language modeling:
print("Training and generating...")
for epoch in range(1, config.num_epochs + 1):
total_loss = 0.0
model.train()
hidden = model.init_hidden(config.batch_size)
for ibatch, i in enumerate(range(0, train_len - 1, seq_len)):
data, targets = get_batch(train_data, i, seq_len)
hidden = repackage_hidden(hidden)
model.zero_grad()
output, hidden = model(data, hidden)
loss = criterion(output.view(-1, config.vocab_size), targets)
loss.backward()
Please check line 5.
And the init_hidden function is as follows:
def init_hidden(self, bsz):
weight = next(self.parameters()).data
if self.rnn_type == 'LSTM': # lstm:(h0, c0)
return (Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()),
Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()))
else: # gru & rnn:h0
return Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_())
My question is:
Why do we need to init_hidden every epoch? Shouldn't it be that the model inherit the hidden parameters from last epoch and continue training on them.