Questions tagged [attention-model]

Questions regarding attention model mechanism in deep learning

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

visualize heatmap of custom layer keras

I have been trying to visualize the heatmap of the custom layer, Like I have the channel and spatial attention and I want the map of both attentions kinda like in this picture Attention map. I got the ...
  • 5
0 votes
0 answers
13 views

Tensorflow Attention ValueError: Dimension must be 5 but is 4

I am trying to follow the below code for a self-attention model. The self-attention networks have 16 heads, and the output of each head is 16-dimensional. The dimension of the additive attention query ...
  • 1
2 votes
0 answers
74 views

Segmentation fault (core dumped) with libpatches.so

Edit3: Loaded core into gdb. Edit2: Included the .cc code. Edit1: loaded debug symbols. I'm trying to run the example mnist program of the attention-sampling github library. The error out put is as ...
1 vote
1 answer
45 views

How to read a BERT attention weight matrix?

I have extracted from the last layer and the last attention head of my BERT model the attention score/weights matrix. However I am not too sure how to read them. The matrix is the following one. I ...
  • 340
2 votes
0 answers
31 views

In the sequential recommendation model TiSASRec, the results of the baseline model SASRec are inconsisent with the actual?

I am a novice in the recommender system. Recently, I was reading a paper related to sequential recommendation. In the process of running the official sample code of TiSASRec, I used the dataset given ...
1 vote
0 answers
11 views

Attention mechanism with different layer size?

i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how? ...
2 votes
0 answers
25 views

How to correctly load a deep learning model with a personalized layer?

I trained and tested a DL model that uses a different version of the Attention layer provided by keras because it needs to do more stuff in respect the basic one. My perfomance are about 90.4% of ...
0 votes
0 answers
31 views

Using Self-Attention Layer in Keras without Encoder-Decoder

Attention has been used with encoder decoder to my knowledge. I am trying to use it as a layer in a feedforward neural network. I have the following archeticturE: Input layer -> Dense Layer -> ...
  • 419
0 votes
1 answer
29 views

Layernorm in PyTorch

Consider the following example: batch, sentence_length, embedding_dim = 2, 3, 4 embedding = torch.randn(batch, sentence_length, embedding_dim) print(embedding) # Output: tensor([[[...
  • 415
1 vote
1 answer
109 views

Difference between MultiheadAttention and Attention layer in Tensorflow

What is the difference between the following layers in Tensorflow: tf.keras.layers.Attention, tf.keras.layers.MultiHeadAttention and tf.keras.layers.AdditiveAttention? Also how to implement tf.keras....
user avatar
0 votes
1 answer
61 views

How Seq2Seq Context Vector is generated?

I have studied the theory of seq2seq model but I couldn't clearly understand that what exactly is context vector and how is it generated. I know it summarizes the meaning of to-be-encoded sequence ...
0 votes
0 answers
17 views

Implement human action recognition using graph attention network

I want to implement human action recognition using graph attention network and here is my code: import numpy as np import tensorflow as tf from tensorflow import keras import pandas as pd import ...
0 votes
0 answers
23 views

pytorch MultiheadAttention - when and where can one use weights (the second output)?

I've read through a few MultiheadAttention tutorials now, and I consistently see the weights return value being ignored, ie: x, _ = myattention(q,k,v) I've also seen a need_weights parameter that can ...
  • 1,153
0 votes
0 answers
57 views

Python code of CNN with attention layer giving error in Keras

I am not that expert in Python. I have created the following code for CNN for image classification with attention layer. The source code is attached below with error. I am using Jupyter Notebook in VS ...
0 votes
0 answers
28 views

Positional Embedding in Transformers - Time Series Data

I'm adding Multi-Headed attention at the input of my CNN to improve interpretability and explainability of my model. The data is formed as time-series 3D input of shape (125, 5, 6) where 5x6 part ...
0 votes
0 answers
19 views

Encoder Decoder with Teacher forcing and attention combine keras

Teacher forcing and attention combine I googled a lot but could not find tutorial that is implementing encoder decoder with teacher forcing and attention such as luong attention. I am confused how ...
0 votes
0 answers
124 views

How to add a multihead attention layer to a CNN-LSTM model?

I'm trying to make a hybrid binary text classification model using a multi-head attention mechanism with CNN-LSTM. However, I'm facing an issue when trying to pass the values obtained from CNN-LSTM to ...
1 vote
0 answers
33 views

Using attention in a CNN attention model

I am very new to python and ML. I have a working CNN LSTM model as below: def model_demo(): inp = Input(shape=(6000,3), name='input') e = Conv1D(16, 9 , strides =1, padding = 'same', ...
0 votes
0 answers
136 views

Apply `torch.nn.MultiheadAttention`’s heads to same input

My question surely has a simple answer, but I couldn't find it. I wish to apply MultiheadAttention to the same sequence without copying the sequence. My data is temporal data with dimensions (batch, ...
  • 1,520
0 votes
1 answer
36 views

Does this code use soft attention or hard attention?

I'm new to deep learning. May I ask if the code below uses soft attention or hard attention? class AttentionBlock(nn.Module): def __init__(self, f_g, f_l, f_int): super().__init__() ...
  • 1
0 votes
0 answers
46 views

Sorting a list of arbitrary size using attention / transformers?

Seq2Seq neural network architectures can work with sequences of arbitrary size either via iteration, as in RNN, or parallelism, as in Transformers or other Attention (Query/Key/Value) mechanisms. It ...
  • 627
0 votes
0 answers
46 views

Attention in Keras

I'm trying to add an attention layer to improve interpretability of my multi-label classification deep learning model. The attention layer is intended to be at the very input into the network and to ...
0 votes
0 answers
33 views

How do I implement a model that finds a correlation (not similarity) between query and target sentence?

When building an NLP model (I'm going to use an attention-based one), how can we implement one for finding the correlation, not similarity, between the query and target sentences? For instance, the ...
  • 1,031
0 votes
0 answers
65 views

What makes the multi-head self-attention matrices different?

Transformers (BERT) use one set of three matrices, Q, K, V for each attention head. BERT uses 12 attention heads in each layer, with each attention head having it's own set of three such matrices. The ...
  • 9
2 votes
1 answer
143 views

Have I implemented self-attention correctly in Pytorch?

This is my attempt at implementing self-attention using PyTorch. Have I done anything wrong, or could it be improved somehow? class SelfAttention(nn.Module): def __init__(self, embedding_dim): ...
0 votes
0 answers
58 views

How to implement attention matrix from pretrained BERT uncased transformer model to analyze IMDB sentiment tasks

I was following this well-known paper and wanted to generate the attention matrices demonstrated on the last three pages of this paper. The data and model I built are based on this extensive tutorial. ...
0 votes
0 answers
27 views

Implementing attention mechanism for neural machine translation

class Attention(nn.Module): def __init__(self, hidden_size): super(Attention, self).__init__() self.hidden_size = hidden_size # Create a two layer fully-connected network. ...
0 votes
1 answer
52 views

Why does the query in multihead attention not affect the output?

I am experimenting with multihead attention and am trying to understand why the value of the query embeddings in the following code has no impact on the attention output, as well as why the output is ...
0 votes
0 answers
24 views

Keras error on Inter-attention implementation "Invalid argument: required broadcastable shapes"

I want to implement Python code using Keras and Tensorflow to create Model prediction using this paper https://www.semanticscholar.org/paper/Bi-ISCA%3A-Bidirectional-Inter-Sentence-Contextual-in-Nadar/...
0 votes
0 answers
29 views

Receptive Field in Swin Transformer

I want to ask if is it true that the receptive field of swin transformer is just in the local window where we compute the self-attention? And is there any way to increase the receptive field when ...
2 votes
1 answer
52 views

Mismatch between computational complexity of Additive attention and RNN cell

According to Attention is all you need paper: Additive attention (The classic attention use in RNN by Bahdanau) computes the compatibility function using a feed-forward network with a single hidden ...
1 vote
1 answer
158 views

Tensorflow Multi Head Attention on Inputs: 4 x 5 x 20 x 64 with attention_axes=2 throwing mask dimension error (tf 2.11.0)

The expectation here is that the attention is applied on the 2nd dimension (4, 5, 20, 64). I am trying to apply self attention using the following code (issue reproducible with this code): import ...
0 votes
1 answer
106 views

How do I mask for 2-D MultiHeadAttention in Tensorflow?

Can anyone help me understand masking a 3D input (technically 4D) in MultiHeadAttention? My original dataset consists of timeseries in the form of: Inputs: (samples, horizon, features) ~> (8, 4, 2) ...
-1 votes
1 answer
40 views

reshaping tensors for multi head attention in pytorch - view vs transpose

I'm learning about the attention operator in the deep learning domain. I understand that to compute multi head attention efficiently in parallel, the input tensors (query, key, value) have to be ...
  • 779
1 vote
2 answers
933 views

Understanding dimensions in MultiHeadAttention layer of Tensorflow

I'm learning multi-head attention with this article. As the writer claimed, the structure of MHA (by the original paper) is as follows: But the MultiHeadAttention layer of Tensorflow seems to be more ...
0 votes
0 answers
417 views

Training deep learning model with sparse matrix for Pytorch

I have a list of data features with variable lengths: X_train[0]: <9x15466 sparse matrix of type '<class 'numpy.float64'>' with 115 stored elements in COOrdinate format> X_train[1]: &...
  • 2,926
0 votes
2 answers
485 views

How to get attention weights from attention neural network?

I have a model that uses an attention mechanism as below: def create_model(feature_size, max_features, num_class): feature_input = Input((max_features,feature_size), dtype=tf.float32) ...
1 vote
0 answers
37 views

Multihead Attention with for loop

According to the Attention is All You Need paper Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the ...
  • 604
0 votes
1 answer
431 views

Adding Luong attention Layer to CNN

I'm using keras to implement a functional CNN model where I have images with the size of 64x64x1. with 6 convolutional layer like this : num_classes = 5 def get_model(): ##creating CNN functional ...
0 votes
1 answer
468 views

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! when using transformers architecture

I am having a multi-gpu problem while practicing transformer through pytorch.All the training previously studied using pytorch was possible just by putting nn.dataparallel on the model object.However, ...
0 votes
0 answers
26 views

Simple self-Attention API to learn from vector sequence

I wanted to implement simple softmax-based self-attention for a sequence of vectors. Using PyTorch's multi-head self-attention API seems overwhelming for my task with a large number of parameters to ...
0 votes
0 answers
27 views

how to check transformer model's attention?

I am doing a project on text summarization. I am also showing the attention mask for training. model is working well but I want to check and show those words on which during training model is giving ...
-3 votes
1 answer
99 views

Difference between Model(inputs=[input],outputs=[output1,output2]) and Model(inputs=[input],outputs=[output1]+output2) in KERAS?

Please check out the last line of the code
0 votes
0 answers
119 views

Next sentence prediction / question answering WITHOUT token_type_ids?

Apparently, BERT tokenizer returns input_ids, attention_mask and token_type_ids, where token_type_ids are used if we have NSP or QA task (it is used to differentiate between the Context and the aspect ...
0 votes
1 answer
384 views

How to use pytorch multi-head attention for classification task?

I have a dataset where x shape is (10000, 102, 300) such as ( samples, feature-length, dimension) and y (10000,) which is my binary label. I want to use multi-head attention using PyTorch. I saw the ...
0 votes
0 answers
269 views

Confusion regarding num_heads & key_dim keras.layers.MultiHeadAttention in the transformer tutorial

In the tf.keras tutorial: https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/transformer.ipynb, class EncoderLayer(tf.keras.layers.Layer): def __init__(self,*, ...
  • 3,471
0 votes
0 answers
26 views

the param of attention layer is 0

when I build multi_head_self_attention ,I found the param of this layer is 0,what is wrong with this attention layer?what should i do to modify this layer? I initialize query, key, value in init,and ...
0 votes
1 answer
74 views

NLP: transformer learning weights

The softmax function obtains the weights and then MatMul with V. Are the weights stored anywhere? Or how the learning process happened if the weights are not stored or used on the next round? Moreover,...
0 votes
1 answer
94 views

Can we use squeeze-and-excite attention block on pretrained network? (only train SENET block, will it work?)

I am using pre-trained VGG19 and applying squeeze-and-excitation attention (SENET) module as mentioned in the Squeeze and Excitaion Network, in between pre-trained VGG19 layer, and only training the ...
0 votes
1 answer
507 views

Tensorflow Training Stops before all epochs are completed, with no error message

I was trying to train a multi-head attention model on some open angle data that I had generated, for a classification problem. The model was meant to test for testing what normalizations to use for my ...

1
2 3 4 5
8