1

If I well understood a RNN allows to predict the next value considering the last values in a sequence. Let say I want to predict the next value of the function cos(x) and I have a dataset with the results for x in range(0, 1000). First I feed the model with cos(0) to predict cos(1), then cos(1) to predict cos(2) etc.. and at each step the weights are tuned and the model keep a memory of the last values to make the next prediction.

In my case I want to train a model to predict the quality of videos. For this I have a dataset with videos annotated. For each video, for each frame, I compute a set of 36 features which are not spatially related. So the shape of the inputs is (nb_videos, nb_frames, 36). For each video I have a score representing the global video quality and the shape of the labels is (nb_videos, 1).

I don't know which kind of NN I can use. n_frames x 36 is far too big I think for a simple multi-layer perceptron. Features may make sense along time axis but not along features axis, so unless I train 36 models with 1D convolutions a CNN is useless. Finally, features come in a sequence but the problem with a RNN is that it need a score for each element of the sequence and the model works only to predict the next values in this sequence in particular.

My idea is to have 1 RNN model which is trained for any video. I feed the RNN n_frames times with the 36 features in the good order and only after these n_frames iterations the model give a prediction. Then this prediction is used to tune the weights. Then we do this the number of epochs with a video picked randomly in the dataset.

Does it make sense?

Does something similar exist?

1 Answer 1

1

I don't think you're making an unconventional use of a RNN/LSTM, and your idea makes sense. If I understood it correctly, your idea involves using a many to one RNN:

Many to one RNN Source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/

where the input at each timestep corresponds to one frame with 36 features, and the output at the last timestep conveys information about the whole video. In Keras, this could be something along the lines of:

from keras.models import Sequential
from keras.layers import LSTM, Dense

nb_frames = 10

model = Sequential()
model.add(LSTM(20, input_shape=(nb_frames, 36)))
model.add(Dense(1, activation='relu'))
model.compile('rmsprop', 'mse')
model.summary()

Many to one RNNs are very common, and you wouldn't be making an unconventional use of them.

4
  • Thanks a lot, that's exactly what I was looking for. What happen to the values stored into the cells between 2 videos?
    – amarion
    Jul 4, 2018 at 9:41
  • You're welcome. I think I don't understand your question. Are you talking about hidden state of the LSTM (the green boxes from the image above)?
    – rvinas
    Jul 4, 2018 at 9:50
  • Yes, I'm talking about the memory in each layer. I suppose that in a batch, at each step, the hidden state depend of the previous step, so in my case the previous frame. But between 2 videos? The hidden state for the first frame/step of the next video/batch depend of the hidden state for the last frame/step of the previous video/batch or is reset at each video/batch?
    – amarion
    Jul 5, 2018 at 7:31
  • 1
    The hidden states of different videos are independent (default behavior in Keras), unless you set the argument stateful to True (see keras.io/layers/recurrent/#lstm)
    – rvinas
    Jul 5, 2018 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.