2

I am writing a sequence to sequence neural network in Pytorch. In the official Pytorch seq2seq tutorial, there is code for an Attention Decoder that I cannot understand/think might contain a mistake.

It computes the attention weights at each time step by concatenating the output and the hidden state at this time, and then multiplying by a matrix to get a vector of size equal to the output sequence length. Note, these attention weights don’t depend on the encoder sequence (named encoder_outputs in the code), which I think it should.

Also, the paper cited in the tutorial, lists three different score functions that can be used to compute attention weights (section 3.1 in the paper). None of these functions is just concatenating and multiplying by a matrix.

So it seems to me that the code in the tutorial is mistaken both in the function it applies and the arguments that are passed to this function. Am I missing something?

2
  • 1
    Read section 3 equation 5
    – papabiceps
    Commented May 2, 2019 at 19:57
  • As I understand it, that equation says what to do with the attention representation, which they call the context vector, denoted c_t, once we get it. See, e.g. the description under figure 2.
    – ludog
    Commented May 2, 2019 at 20:45

2 Answers 2

0

This tutorial has a simplified version of these attentions in the Luong paper that you mentioned.

It just uses a linear layer to combine the input embedding and the decoder RNN hidden state. This is sometimes called a 'location-based' attention, because it does not depend on the encoder outputs. Then it applies the softmax and computes the attention weights and the process goes as it would normally.

This is not always bad to have, as from the encoder outputs the attention mechanism might attend to a previous token and then the attention would not be monotonic, so your model would fail.

To implement the attentions from the Luong paper, you I suggest to use the 'concat' attention, after applying linear layers to both the decoder hidden state and the encoder outputs. Then the matrix W_a will transform these concatenated results to an arbitrary dimension of your choice, and finally the v_a is a vector that will transform to the desired context vector dimension.

4
  • Hey, thanks for the reply. Is this simplified attention technique described in a paper anywhere? It seems strange that pytorch would make up their own new attention mechanism and not mention that they had done so.
    – ludog
    Commented Dec 6, 2019 at 17:31
  • I don't exactly remember if I have seen a paper doing this (if there is, it should definitely be an NLP paper), but it was an earlier practice, also for self-attention. Instead of doing the formula from the 'Attention is all you need' paper, they just put a linear layer on top of the encoder to learn the attention weights!
    – njellinas
    Commented Dec 11, 2019 at 14:39
  • Hmm, it still seems strange to me that the weights would not depend on the encoder outputs. It kind of challenges the typical intuition given for the success of attention mechanisms. If you have any resources on this, for example where else you have seen it implemented, I would be interested to see them.
    – ludog
    Commented Dec 11, 2019 at 20:55
  • Yes, look at this paper at page 26 arxiv.org/pdf/1308.0850.pdf
    – njellinas
    Commented Dec 12, 2019 at 11:16
0

In the algorithm, attn_weights depends on decode parameters. Then we get an output of a linear layer(here 10). This is attention vector. Then we multiply this with encoder_outputs. So at every epoch, we update attn_weights by back propagation. Verbally, at every iteration, it is learning in the reverse direction. Let me give an example:

Our task is translate from English to German.

I want to sing a song. -> Ich möchte ein Lied singen.

At decoder, singen verb is at end. So our decoder attn_weights see decoder output,and learns to apply which parts of input encoding. When you multiply this value with encoder_outputs , you get a matrix of ,which have high values in necessary points. So infact this way, it is learning when decoder see a sentence pattern in german, which parts of input it must pay attention. So direction of learning is correct,I think.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.