1

I have the following code:

import torch
import torch.nn as nn

model = nn.Sequential(
          nn.LSTM(300, 300),
          nn.Linear(300, 100),
          nn.ReLU(),
          nn.Linear(300, 7),
          )

s = torch.ones(1, 50, 300)
a = model(s)

And I get:

My-MBP:Desktop myname$ python3 testmodel.py 
Traceback (most recent call last):
  File "testmodel.py", line 12, in <module>
    a = model(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/functional.py", line 1688, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

Why? The dimensions should be fine. I saw related fixes to this issue when *input is defined in model.forward, but I don't even have anything implemented yet.

/edit: WAIT, there IS a *input!? How can I override this?

1
  • 1
    This error is because the output of an LSTM is a tuple containing the output, hidden state, and cell state. You cannot pass this into a linear layer. You are only supposed to pass in the output of the LSTM not the hidden and cell states too. Jan 26, 2021 at 18:51

2 Answers 2

3

You won't be able to use a nn.RNN inside a nn.Sequential since nn.LSTM layers will output a tuple containing (1) the output features and (2) hidden states and cell states.

The output must first be unpacked in order to use the output features in your subsequent layer: nn.Linear. Something as, if your interested in the hidden states and cell states:

rnn = nn.LSTM(300, 300)
output, (h_n, c_n) = rnn(x)

You could define a custom nn.Module and implement a simple forward function:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.rnn = nn.LSTM(300, 300)
        
        self.body = nn.Sequential(
          nn.Linear(300, 100),
          nn.ReLU(),
          nn.Linear(100, 7)) # <- had it set to in_features=300

    def forward(self, x):
        x, _ = self.rnn(x) # <- ignore second output
        x = self.body(x)
        return x

Such that:

>>> model = Model()
>>> s = torch.ones(1, 50, 300)

>>> model(s).shape
torch.Size([1, 50, 7])
2
  • Thank you! I know I had three outputs, but somehow I didn't come to mind this may cause troubles. Quick follow up question: I want to use an encoder architecture for text classification. My [1, 50, 300] is a sentence of length 50 where each word is embedded with dimension 300. The encoded sequence shall be passed to the FC layer. Do I just take the last 'row' of the lstm tensor and pass it?, i.e. (not very beautiful): rnn(s)[0][-1][-1].reshape(1,1,300) Jan 27, 2021 at 11:56
  • 1
    If 1 is your batch size, then you definitely need to set batch_first=True on your nn.LSTM. So your RNN output will be (batch, seq_len, input_size, hidden_size), in your case: (1, 50, 300). Using rnn(s)[0][-1][-1] will select the prediction on the last timestep of the last batch element. Instead use rnn(s)[0][:, -1]. A cleaner way (imo) is to unpack first (c.f above) then take [:, -1] from the first tuple element.
    – Ivan
    Jan 27, 2021 at 12:33
3

The error is because nn.LSTM returns your output and your model's state, which is a tuple containing the hidden state and the memory state.

You can fix it by defining your own nn.Module class that returns just the output of the LSTM for you.

class GetLSTMOutput(nn.Module):
    def forward(self, x):
        out, _ = x
        return out

model = nn.Sequential(
    nn.LSTM(300, 300),
    GetLSTMOutput(),
    nn.Linear(300, 100),
    nn.ReLU(),
    nn.Linear(300, 7))

The forward method of the GetLSTMOutput class is call implicitly and the output of the previous layer nn.LSTM(300, 300) is passed as an argument. It then returns just the output part, discarding the model's state.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.