1

I am using the LSTM cell in Tensorflow.

lstm_cell = tf.contrib.rnn.BasicLSTMCell(lstm_units)

I was wondering how the weights and states are initialized or rather what the default initializer is for LSTM cells (states and weights) in Tensorflow?

And is there an easy way to manually set an Initializer?

Note: For tf.get_variable() the glorot_uniform_initializer is used as far as I could find out from the documentation.

2 Answers 2

7

First of all, there is a difference between the weights of a LSTM (the usual parameter set of a ANN), which are by default also initialized by the Glorot or also known as the Xavier initializer (as mentioned in the question).

A different aspect is the cell state and the state of the initial recurrent input to the LSTM. Those are initialized by a matrix usually denoted as initial_state.

Leaving us with the question, how to initialize this initial_state:

  1. Zero State Initialization is good practice if the impact of initialization is low

The default approach to initializing the state of an RNN is to use a zero state. This often works well, particularly for sequence-to-sequence tasks like language modeling where the proportion of outputs that are significantly impacted by the initial state is small.

  1. Zero State Initialization in each batch can lead to overfitting

Zero Initialization for each batch will lead to the following: Losses at the early steps of a sequence-to-sequence model (i.e., those immediately after a state reset) will be larger than those at later steps, because there is less history. Thus, their contribution to the gradient during learning will be relatively higher. But if all state resets are associated with a zero-state, the model can (and will) learn how to compensate for precisely this. As the ratio of state resets to total observations increases, the model parameters will become increasingly tuned to this zero state, which may affect performance on later time steps.

  1. Do we have other options?

One simple solution is to make the initial state noisy (to decrease the loss for the first time step). Look here for details and other ideas

0

I don't think you can initialize an individual cell, but when you execute the LSTM with tf.nn.static_rnn or tf.nn.dynamic_rnn, you can set the initial_state argument to a tensor containing the LSTM's initial values.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.