20

I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector or return_seq=True. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one?

First model using return_sequence=True:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)

encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)

autoencoder = Model(inputEncoder, outDecoder)

Second model with RepeatVector:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)

firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)

outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)

autoencoder = Model(inputEncoder, outDecoder)
1
  • I am curios, what's the reason for RepeatVector(1) in your code. Are you just using it to add a time dimension of 1? But then you follow it up with Reshape((28, 1)) and take it right out... Or did I misunderstand something in your code? Aug 30, 2020 at 14:34

1 Answer 1

71

You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches.

Difference <code>return_sequences=True</code> and RepeatVector Essentially, return_sequences=True returns all the outputs the encoder observed in the past, while RepeatVector repeats the very last output of the encoder.

8
  • 1
    Open up my eyes, now I can see better :))
    – sariii
    Jun 3, 2019 at 23:11
  • 1
    Again thanks for the insightful answer, do you have any idea where we should use which? Im not saying for 100% but like generally!
    – sariii
    Jun 3, 2019 at 23:16
  • 3
    Hi @sariii, one example is machine translation. For example if you have a seq2seq model and you don't want to use teacher forcing and want a quick and dirty solution, you can pass in the last state of the encoder RNN (last blue box) using RepeatVector and return_sequences=False. But if you are to compute attention weights for the encoder states, you will need to use return_sequences=True as you need all encoder states to compute attention weights. Hope that makes sense. :)
    – thushv89
    Jun 6, 2019 at 23:07
  • @thushv89 Thank you so much for the explanations. Can you please do a favor for me and have a look at this question? stackoverflow.com/questions/56433993/… I have read a lot, even know what is the problem but unable to fix it
    – sariii
    Jun 6, 2019 at 23:11
  • 2
    @Coderji, I created them. I didn't get it from anywhere :)
    – thushv89
    Jun 8, 2020 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.