Questions tagged [attention-model]
Questions regarding attention model mechanism in deep learning
381
questions
0
votes
0
answers
13
views
visualize heatmap of custom layer keras
I have been trying to visualize the heatmap of the custom layer, Like I have the channel and spatial attention and I want the map of both attentions kinda like in this picture Attention map. I got the ...
0
votes
0
answers
13
views
Tensorflow Attention ValueError: Dimension must be 5 but is 4
I am trying to follow the below code for a self-attention model.
The self-attention networks have 16 heads, and the output of each head is 16-dimensional. The dimension of the additive attention query ...
2
votes
0
answers
74
views
Segmentation fault (core dumped) with libpatches.so
Edit3: Loaded core into gdb.
Edit2: Included the .cc code.
Edit1: loaded debug symbols.
I'm trying to run the example mnist program of the attention-sampling github library.
The error out put is as ...
1
vote
1
answer
45
views
How to read a BERT attention weight matrix?
I have extracted from the last layer and the last attention head of my BERT model the attention score/weights matrix. However I am not too sure how to read them. The matrix is the following one. I ...
2
votes
0
answers
31
views
In the sequential recommendation model TiSASRec, the results of the baseline model SASRec are inconsisent with the actual?
I am a novice in the recommender system. Recently, I was reading a paper related to sequential recommendation. In the process of running the official sample code of TiSASRec, I used the dataset given ...
1
vote
0
answers
11
views
Attention mechanism with different layer size?
i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how?
...
2
votes
0
answers
25
views
How to correctly load a deep learning model with a personalized layer?
I trained and tested a DL model that uses a different version of the Attention layer provided by keras because it needs to do more stuff in respect the basic one. My perfomance are about 90.4% of ...
0
votes
0
answers
31
views
Using Self-Attention Layer in Keras without Encoder-Decoder
Attention has been used with encoder decoder to my knowledge. I am trying to use it as a layer in a feedforward neural network. I have the following archeticturE:
Input layer -> Dense Layer -> ...
0
votes
1
answer
29
views
Layernorm in PyTorch
Consider the following example:
batch, sentence_length, embedding_dim = 2, 3, 4
embedding = torch.randn(batch, sentence_length, embedding_dim)
print(embedding)
# Output:
tensor([[[...
1
vote
1
answer
109
views
Difference between MultiheadAttention and Attention layer in Tensorflow
What is the difference between the following layers in Tensorflow: tf.keras.layers.Attention, tf.keras.layers.MultiHeadAttention and tf.keras.layers.AdditiveAttention?
Also how to implement tf.keras....
0
votes
1
answer
61
views
How Seq2Seq Context Vector is generated?
I have studied the theory of seq2seq model but I couldn't clearly understand that what exactly is context vector and how is it generated. I know it summarizes the meaning of to-be-encoded sequence ...
0
votes
0
answers
17
views
Implement human action recognition using graph attention network
I want to implement human action recognition using graph attention network and here is my code:
import numpy as np
import tensorflow as tf
from tensorflow import keras
import pandas as pd
import ...
0
votes
0
answers
23
views
pytorch MultiheadAttention - when and where can one use weights (the second output)?
I've read through a few MultiheadAttention tutorials now, and I consistently see the weights return value being ignored, ie:
x, _ = myattention(q,k,v)
I've also seen a need_weights parameter that can ...
0
votes
0
answers
57
views
Python code of CNN with attention layer giving error in Keras
I am not that expert in Python. I have created the following code for CNN for image classification with attention layer. The source code is attached below with error. I am using Jupyter Notebook in VS ...
0
votes
0
answers
28
views
Positional Embedding in Transformers - Time Series Data
I'm adding Multi-Headed attention at the input of my CNN to improve interpretability and explainability of my model. The data is formed as time-series 3D input of shape (125, 5, 6) where 5x6 part ...
0
votes
0
answers
19
views
Encoder Decoder with Teacher forcing and attention combine keras
Teacher forcing and attention combine
I googled a lot but could not find tutorial that is implementing encoder decoder with teacher forcing and attention such as luong attention. I am confused how ...
0
votes
0
answers
124
views
How to add a multihead attention layer to a CNN-LSTM model?
I'm trying to make a hybrid binary text classification model using a multi-head attention mechanism with CNN-LSTM. However, I'm facing an issue when trying to pass the values obtained from CNN-LSTM to ...
1
vote
0
answers
33
views
Using attention in a CNN attention model
I am very new to python and ML.
I have a working CNN LSTM model as below:
def model_demo():
inp = Input(shape=(6000,3), name='input')
e = Conv1D(16, 9 , strides =1, padding = 'same', ...
0
votes
0
answers
136
views
Apply `torch.nn.MultiheadAttention`’s heads to same input
My question surely has a simple answer, but I couldn't find it. I wish to apply MultiheadAttention to the same sequence without copying the sequence. My data is temporal data with dimensions (batch, ...
0
votes
1
answer
36
views
Does this code use soft attention or hard attention?
I'm new to deep learning. May I ask if the code below uses soft attention or hard attention?
class AttentionBlock(nn.Module):
def __init__(self, f_g, f_l, f_int):
super().__init__()
...
0
votes
0
answers
46
views
Sorting a list of arbitrary size using attention / transformers?
Seq2Seq neural network architectures can work with sequences of arbitrary size either via iteration, as in RNN, or parallelism, as in Transformers or other Attention (Query/Key/Value) mechanisms. It ...
0
votes
0
answers
46
views
Attention in Keras
I'm trying to add an attention layer to improve interpretability of my multi-label classification deep learning model. The attention layer is intended to be at the very input into the network and to ...
0
votes
0
answers
33
views
How do I implement a model that finds a correlation (not similarity) between query and target sentence?
When building an NLP model (I'm going to use an attention-based one), how can we implement one for finding the correlation, not similarity, between the query and target sentences?
For instance, the ...
0
votes
0
answers
65
views
What makes the multi-head self-attention matrices different?
Transformers (BERT) use one set of three matrices, Q, K, V for each attention head. BERT uses 12 attention heads in each layer, with each attention head having it's own set of three such matrices.
The ...
2
votes
1
answer
143
views
Have I implemented self-attention correctly in Pytorch?
This is my attempt at implementing self-attention using PyTorch. Have I done anything wrong, or could it be improved somehow?
class SelfAttention(nn.Module):
def __init__(self, embedding_dim):
...
0
votes
0
answers
58
views
How to implement attention matrix from pretrained BERT uncased transformer model to analyze IMDB sentiment tasks
I was following this well-known paper and wanted to generate the attention matrices demonstrated on the last three pages of this paper.
The data and model I built are based on this extensive tutorial. ...
0
votes
0
answers
27
views
Implementing attention mechanism for neural machine translation
class Attention(nn.Module):
def __init__(self, hidden_size):
super(Attention, self).__init__()
self.hidden_size = hidden_size
# Create a two layer fully-connected network. ...
0
votes
1
answer
52
views
Why does the query in multihead attention not affect the output?
I am experimenting with multihead attention and am trying to understand why the value of the query embeddings in the following code has no impact on the attention output, as well as why the output is ...
0
votes
0
answers
24
views
Keras error on Inter-attention implementation "Invalid argument: required broadcastable shapes"
I want to implement Python code using Keras and Tensorflow to create Model prediction using this paper https://www.semanticscholar.org/paper/Bi-ISCA%3A-Bidirectional-Inter-Sentence-Contextual-in-Nadar/...
0
votes
0
answers
29
views
Receptive Field in Swin Transformer
I want to ask if is it true that the receptive field of swin transformer is just in the local window where we compute the self-attention? And is there any way to increase the receptive field when ...
2
votes
1
answer
52
views
Mismatch between computational complexity of Additive attention and RNN cell
According to Attention is all you need paper: Additive attention (The classic attention use in RNN by Bahdanau) computes the compatibility function using a feed-forward network with a single hidden ...
1
vote
1
answer
158
views
Tensorflow Multi Head Attention on Inputs: 4 x 5 x 20 x 64 with attention_axes=2 throwing mask dimension error (tf 2.11.0)
The expectation here is that the attention is applied on the 2nd dimension (4, 5, 20, 64). I am trying to apply self attention using the following code (issue reproducible with this code):
import ...
0
votes
1
answer
106
views
How do I mask for 2-D MultiHeadAttention in Tensorflow?
Can anyone help me understand masking a 3D input (technically 4D) in MultiHeadAttention?
My original dataset consists of timeseries in the form of:
Inputs: (samples, horizon, features) ~> (8, 4, 2) ...
-1
votes
1
answer
40
views
reshaping tensors for multi head attention in pytorch - view vs transpose
I'm learning about the attention operator in the deep learning domain. I understand that to compute multi head attention efficiently in parallel, the input tensors (query, key, value) have to be ...
1
vote
2
answers
933
views
Understanding dimensions in MultiHeadAttention layer of Tensorflow
I'm learning multi-head attention with this article.
As the writer claimed, the structure of MHA (by the original paper) is as follows:
But the MultiHeadAttention layer of Tensorflow seems to be more ...
0
votes
0
answers
417
views
Training deep learning model with sparse matrix for Pytorch
I have a list of data features with variable lengths:
X_train[0]:
<9x15466 sparse matrix of type '<class 'numpy.float64'>'
with 115 stored elements in COOrdinate format>
X_train[1]:
&...
0
votes
2
answers
485
views
How to get attention weights from attention neural network?
I have a model that uses an attention mechanism as below:
def create_model(feature_size, max_features, num_class):
feature_input = Input((max_features,feature_size), dtype=tf.float32)
...
1
vote
0
answers
37
views
Multihead Attention with for loop
According to the Attention is All You Need paper
Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the ...
0
votes
1
answer
431
views
Adding Luong attention Layer to CNN
I'm using keras to implement a functional CNN model where I have images with the size of 64x64x1. with 6 convolutional layer like this :
num_classes = 5
def get_model():
##creating CNN functional ...
0
votes
1
answer
468
views
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! when using transformers architecture
I am having a multi-gpu problem while practicing transformer through pytorch.All the training previously studied using pytorch was possible just by putting nn.dataparallel on the model object.However, ...
0
votes
0
answers
26
views
Simple self-Attention API to learn from vector sequence
I wanted to implement simple softmax-based self-attention for a sequence of vectors. Using PyTorch's multi-head self-attention API seems overwhelming for my task with a large number of parameters to ...
0
votes
0
answers
27
views
how to check transformer model's attention?
I am doing a project on text summarization. I am also showing the attention mask for training. model is working well but I want to check and show those words on which during training model is giving ...
-3
votes
1
answer
99
views
Difference between Model(inputs=[input],outputs=[output1,output2]) and Model(inputs=[input],outputs=[output1]+output2) in KERAS?
Please check out the last line of the code
0
votes
0
answers
119
views
Next sentence prediction / question answering WITHOUT token_type_ids?
Apparently, BERT tokenizer returns input_ids, attention_mask and token_type_ids,
where token_type_ids are used if we have NSP or QA task (it is used to differentiate between the Context and the aspect ...
0
votes
1
answer
384
views
How to use pytorch multi-head attention for classification task?
I have a dataset where x shape is (10000, 102, 300) such as ( samples, feature-length, dimension) and y (10000,) which is my binary label. I want to use multi-head attention using PyTorch. I saw the ...
0
votes
0
answers
269
views
Confusion regarding num_heads & key_dim keras.layers.MultiHeadAttention in the transformer tutorial
In the tf.keras tutorial: https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/transformer.ipynb,
class EncoderLayer(tf.keras.layers.Layer):
def __init__(self,*,
...
0
votes
0
answers
26
views
the param of attention layer is 0
when I build multi_head_self_attention ,I found the param of this layer is 0,what is wrong with this attention layer?what should i do to modify this layer?
I initialize query, key, value in init,and ...
0
votes
1
answer
74
views
NLP: transformer learning weights
The softmax function obtains the weights and then MatMul with V.
Are the weights stored anywhere? Or how the learning process happened if the weights are not stored or used on the next round?
Moreover,...
0
votes
1
answer
94
views
Can we use squeeze-and-excite attention block on pretrained network? (only train SENET block, will it work?)
I am using pre-trained VGG19 and applying squeeze-and-excitation attention (SENET) module as mentioned in the Squeeze and Excitaion Network, in between pre-trained VGG19 layer, and only training the ...
0
votes
1
answer
507
views
Tensorflow Training Stops before all epochs are completed, with no error message
I was trying to train a multi-head attention model on some open angle data that I had generated, for a classification problem. The model was meant to test for testing what normalizations to use for my ...