17

I am using dynamic_rnn to process MNIST data:

# LSTM Cell
lstm = rnn_cell.LSTMCell(num_units=200,
                         forget_bias=1.0,
                         initializer=tf.random_normal)

# Initial state
istate = lstm.zero_state(batch_size, "float")

# Get lstm cell output
output, states = rnn.dynamic_rnn(lstm, X, initial_state=istate)

# Output at last time point T
output_at_T = output[:, 27, :]

Full code: http://pastebin.com/bhf9MgMe

The input to the lstm is (batch_size, sequence_length, input_size)

As a result the dimensions of output_at_T is (batch_size, sequence_length, num_units) where num_units=200.

I need to get the last output along the sequence_length dimension. In the code above, this is hardcoded as 27. However, I do not know the sequence_length in advance as it can change from batch to batch in my application.

I tried:

output_at_T = output[:, -1, :]

but it says negative indexing is not implemented yet, and I tried using a placeholder variable as well as a constant (into which I could ideally feed the sequence_length for a particular batch); neither worked.

Any way to implement something like this in tensorflow atm?

2

7 Answers 7

13

Have you noticed that there are two outputs from dynamic_rnn?

  1. Output 1, let's call it h, has all outputs at each time steps (i.e. h_1, h_2, etc),
  2. Output 2, final_state, has two elements: the cell_state, and the last output for each element of the batch (as long as you input the sequence length to dynamic_rnn).

So from:

h, final_state= tf.dynamic_rnn( ..., sequence_length=[batch_size_vector], ... )

the last state for each element in the batch is:

final_state.h

Note that this includes the case when the length of the sequence is different for each element of the batch, as we are using the sequence_length argument.

3
  • 1
    Does final_state.h return the last output or the last hidden state? Jan 28, 2018 at 12:37
  • 1
    @bluesummers final_state.h is the final activation for each element in the batch (i.e. final output) and final_state.c the state of the cell
    – Escachator
    Jan 28, 2018 at 12:56
  • thanks. Just to be clear, final_state.h will have the dimensions of [batch_size, lstm_units] ? Jan 28, 2018 at 15:09
5

This is what gather_nd is for!

def extract_axis_1(data, ind):
    """
    Get specified elements along the first axis of tensor.
    :param data: Tensorflow tensor that will be subsetted.
    :param ind: Indices to take (one for each element along axis 0 of data).
    :return: Subsetted tensor.
    """

    batch_range = tf.range(tf.shape(data)[0])
    indices = tf.stack([batch_range, ind], axis=1)
    res = tf.gather_nd(data, indices)

    return res

In your case (assuming sequence_length is a 1-D tensor with the length of each axis 0 element):

output = extract_axis_1(output, sequence_length - 1)

Now output is a tensor of dimension [batch_size, num_cells].

9
  • 1
    There is no need to do this, dynamic_rnn has as an output the last state. Check my answer.
    – Escachator
    Aug 6, 2017 at 21:38
  • @Escachator That only works if the sequences are of the same length (inside a batch). OP says "I do not know the sequence_length in advance as it can change from batch to batch in my application" so that may be a fair assumption - I don't know. My answer works even if sequences are of different length inside the batch (and the data is padded).
    – Alex
    Nov 15, 2017 at 15:54
  • if you know (or get) 'ind' you know the sequence length for each element of your batch. Then you can input that in the 'sequence_length' parameter of dynamic_rnn and it will stop calculating once it arrives there on each element of the batch. Then you just take the final_state.h. I possibly should add that to my answer.
    – Escachator
    Nov 15, 2017 at 15:59
  • @Escachator That is true but it doesn't really help if you want to access those last elements! See this post.
    – Alex
    Nov 15, 2017 at 16:00
  • you can always develop a function that, given a batch, gives you the sequence length of each element. I assume you are padding it. You can do that even using the new tf.data.Dataset API with the map function. Then you input that in the sequence_length on the tf.dynamic_rnn and you are done. It could be even faster, as dynamic_rnn will stop calculating on each element once it arrives to the end of it.
    – Escachator
    Nov 15, 2017 at 16:40
3
output[:, -1, :]

works with Tensorflow 1.x now!!

2
  • 1
    This only works if the sequence_length is the same for each element in the output.
    – Alex
    Apr 8, 2017 at 18:23
  • the question is specifically about dynamic lengths, for which this does not work
    – Jules G.M.
    Mar 24, 2018 at 22:58
2

Most answers cover it thoroughly, but this code snip might help understand what's really being returned by the dynamic_rnn layer

=> Tuple of (outputs, final_output_state).

So for an input with max sequence length of T time steps outputs is of the shape [Batch_size, T, num_inputs] (given time_major=False; default value) and it contains the output state at each timestep h1, h2.....hT.

And final_output_state is of the shape [Batch_size,num_inputs] and has the final cell state cT and output state hT of each batch sequence.

But since the dynamic_rnn is being used my guess is your sequence lengths vary for each batch.

    import tensorflow as tf
    import numpy as np
    from tensorflow.contrib import rnn
    tf.reset_default_graph()

    # Create input data
    X = np.random.randn(2, 10, 8)

    # The second example is of length 6 
    X[1,6:] = 0
    X_lengths = [10, 6]

    cell = tf.nn.rnn_cell.LSTMCell(num_units=64, state_is_tuple=True)

    outputs, states  = tf.nn.dynamic_rnn(cell=cell,
                                         dtype=tf.float64,
                                         sequence_length=X_lengths,
                                         inputs=X)

    result = tf.contrib.learn.run_n({"outputs": outputs, "states":states},
                                    n=1,
                                    feed_dict=None)
    assert result[0]["outputs"].shape == (2, 10, 64)
    print result[0]["outputs"].shape
    print result[0]["states"].h.shape
    # the final outputs state and states returned must be equal for each      
    # sequence
    assert(result[0]["outputs"][0][-1]==result[0]["states"].h[0]).all()
    assert(result[0]["outputs"][-1][5]==result[0]["states"].h[-1]).all()
    assert(result[0]["outputs"][-1][-1]==result[0]["states"].h[-1]).all()

The final assertion will fail as the final state for the 2nd sequence is at 6th time step ie. the index 5 and the rest of the outputs from [6:9] are all 0s in the 2nd timestep

1

I am new to Stackoverflow and cannot comment yet so I am writing this new answer. @VM_AI, the last index is tf.shape(output)[1] - 1. So, reusing your answer:

# Let's first fetch the last index of seq length
# last_index would have a scalar value
last_index = tf.shape(output)[1] - 1
# Then let's reshape the output to [sequence_length,batch_size,num_units]
# for convenience
output_rs = tf.transpose(output,[1,0,2])
# Last state of all batches
last_state = tf.nn.embedding_lookup(output_rs,last_index)

This works for me.

1
  • the question asks for the last dynamic state, not the last state in the array
    – Jules G.M.
    Mar 22, 2018 at 2:54
0

You should be able to access the shape of your output tensor using tf.shape(output). The tf.shape() function will return a 1d tensor containing the sizes of the output tensor. In your example, this would be (batch_size, sequence_length, num_units)

You should then be able to extract the value of output_at_T as output[:, tf.shape(output)[1], :]

0

There is a function in TensorFlow tf.shape that allows you to get the symbolic interpretation of shape rather than None being returned by output._shape[1]. And after fetching the last index you can lookup by using tf.nn.embedding_lookup, which is recommended especially when the data to be fetched is high as this does parallel lookup 32 by default.

# Let's first fetch the last index of seq length
# last_index would have a scalar value
last_index = tf.shape(output)[1] 
# Then let's reshape the output to [sequence_length,batch_size,num_units]
# for convenience
output_rs = tf.transpose(output,[1,0,2])
# Last state of all batches
last_state = tf.nn.embedding_lookup(output_rs,last_index)

This should work.

Just to clarify what @Benoit Steiner said. His solution would not work as tf.shape would return symbolic interpretation of the shape value, and such cannot be used for slicing tensors i.e., direct indexing

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.