After reading several articles, I am still quite confused about correctness of my implementation of getting last hidden states from BiLSTM.
- Understanding Bidirectional RNN in PyTorch (TowardsDataScience)
- PackedSequence for seq2seq model (PyTorch forums)
- What's the difference between “hidden” and “output” in PyTorch LSTM? (StackOverflow)
- Select tensor in a batch of sequences (Pytorch formums)
The approach from the last source (4) seems to be the cleanest for me, but I am still uncertain if I understood the thread correctly. Am I using the right final hidden states from LSTM and reversed LSTM? This is my implementation
# pos contains indices of words in embedding matrix
# seqlengths contains info about sequence lengths
# so for instance, if batch_size is 2 and pos=[4,6,9,3,1] and
# seqlengths contains [3,2], we have batch with samples
# of variable length [4,6,9] and [3,1]
all_in_embs = self.in_embeddings(pos)
in_emb_seqs = pack_sequence(torch.split(all_in_embs, seqlengths, dim=0))
output,lasthidden = self.rnn(in_emb_seqs)
if not self.data_processor.use_gru:
lasthidden = lasthidden[0]
# u_emb_batch has shape batch_size x embedding_dimension
# sum last state from forward and backward direction
u_emb_batch = lasthidden[-1,:,:] + lasthidden[-2,:,:]
Is it correct?