I'm new to the tensorflow and trying to implement the "seq2seq" model according to the tutorial. I'm not sure about one argument "num_heads" (default=1) of the func "embedding_attention_seq2seq". What does it represent? I didn't find it in the related papers.
1 Answer
Had you read the source code of any decoder like this one you would get to know that it represents the number of attentions.
Sometimes there are several attentions(hierarchical attentions), for instance this one(as depicted bellow) in this paper.
TL;DR; the first one is for the word and the second one is for the sentence.
Please check this graph:
-
Is that how many attention vectors we calculated for one unit in the decoder? How to change the number of attentions? Is that like we run over encoder states twice? Jun 9, 2017 at 6:08
-
1Yes, you can verify those in the following code, especially this line. Jun 9, 2017 at 6:13
-
yes. But I have this problem.If we have two attentions what will happen? Jun 9, 2017 at 6:16
-
1Perfect. So if we have 2 heads it will calculate attention with two different weight set in-order to capture more information. In some papers I'v seen that calculating attention with different hidden layers in the encoder.Since here attention head parameter is not specifically mentioned I think this is about calculating the attention with two parameters sets. Jun 9, 2017 at 7:21