26

I have a dataset X which consists N = 4000 samples, each sample consists of d = 2 features (continuous values) spanning back t = 10 time steps. I also have the corresponding 'labels' of each sample which are also continuous values, at time step 11.

At the moment my dataset is in the shape X: [4000,20], Y: [4000].

I want to train an LSTM using TensorFlow to predict the value of Y (regression), given the 10 previous inputs of d features, but I am having a tough time implementing this in TensorFlow.

The main problem I have at the moment is understanding how TensorFlow is expecting the input to be formatted. I have seen various examples such as this, but these examples deal with one big string of continuous time series data. My data is different samples, each an independent time series.

1
  • Hi, I am trying to implement something very similar to what you did and am hoping you can give me some tips since tensorflow is still mind boggling for me. For your setup, how does the input file actually look like? Is every sample basically a list of length 10 with each element containing the 2 features and for each sample you have a label? e.g. [[f1,f2], [f1,f2], ...]
    – Dimebag
    Commented Apr 11, 2017 at 8:38

2 Answers 2

20

The documentation of tf.nn.dynamic_rnn states:

inputs: The RNN inputs. If time_major == False (default), this must be a Tensor of shape: [batch_size, max_time, ...], or a nested tuple of such elements.

In your case, this means that the input should have a shape of [batch_size, 10, 2]. Instead of training on all 4000 sequences at once, you'd use only batch_size many of them in each training iteration. Something like the following should work (added reshape for clarity):

batch_size = 32
# batch_size sequences of length 10 with 2 values for each timestep
input = get_batch(X, batch_size).reshape([batch_size, 10, 2])
# Create LSTM cell with state size 256. Could also use GRUCell, ...
# Note: state_is_tuple=False is deprecated;
# the option might be completely removed in the future
cell = tf.nn.rnn_cell.LSTMCell(256, state_is_tuple=True)
outputs, state = tf.nn.dynamic_rnn(cell,
                                   input,
                                   sequence_length=[10]*batch_size,
                                   dtype=tf.float32)

From the documentation, outputs will be of shape [batch_size, 10, 256], i.e. one 256-output for each timestep. state will be a tuple of shapes [batch_size, 256]. You could predict your final value, one for each sequence, from that:

predictions = tf.contrib.layers.fully_connected(state.h,
                                                num_outputs=1,
                                                activation_fn=None)
loss = get_loss(get_batch(Y).reshape([batch_size, 1]), predictions)

The number 256 in the shapes of outputs and state is determined by cell.output_size resp. cell.state_size. When creating the LSTMCell like above, these are the same. Also see the LSTMCell documentation.

4
  • Thanks for the response! I will try this out soon and let you know if it solves my problem. One question though: what exactly does the 256 refer to in tf.nn.rnn_cell.LSTMCell(256, state_is_tuple=True)? I have read the documentation and the value is referred to as n_units. Does that mean number of time steps? ie the memory of the LSTM cell? Sorry, I know this is an extension of the original question. Commented Sep 6, 2016 at 11:33
  • The number of time steps in each sequence is given by the sequence_length parameter that you give into tf.nn.dynamic_rnn. The 256 refers to the size of the internal state of the LSTM which is updated on each time step.
    – fwalch
    Commented Sep 6, 2016 at 12:24
  • @fwalch question: wouldn't you want your fully connected layer for predictions to have num_outputs=batch_size? Then you would have one prediction at the end of each time series in your batch?
    – Engineero
    Commented Apr 25, 2017 at 23:10
  • @Engineero The fully connected layer has output shape [batch_size, num_outputs], i.e. num_outputs is the number of outputs for each entry in your batch.
    – fwalch
    Commented May 4, 2017 at 7:56
5

(This answer "addreses" the problem when direct np.reshape() doesn't organize the final array as we want it. If we want to directly reshape into 3D np.reshape will do it, but watch out for the final organization of the input).

In my personal try to finally resolve this problem of feeding input shape for RNN and not confuse anymore, I will give my "personal" explanation for this.

In my case (and I think that many others may have this organization scheme in their feature matrices), most of the blogs outside "don't help". Let's give it a try in how to convert a 2D feature matrix into a 3D shaped one for RNNs.

Let's say we have this organization type in our feature matrix: we have 5 observations (i.e. rows - for convention I think it is the most logical term to use) and in each row, we have 2 features for EACH timestep (and we have 2 timesteps), like this:

(The df is to better understand visually my words)

In [1]: import numpy as np                                                           

In [2]: arr = np.random.randint(0,10,20).reshape((5,4))                              

In [3]: arr                                                                          
Out[3]: 
array([[3, 7, 4, 4],
       [7, 0, 6, 0],
       [2, 0, 2, 4],
       [3, 9, 3, 4],
       [1, 2, 3, 0]])

In [4]: import pandas as pd                                                          

In [5]: df = pd.DataFrame(arr, columns=['f1_t1', 'f2_t1', 'f1_t2', 'f2_t2'])         

In [6]: df                                                                           
Out[6]: 
   f1_t1  f2_t1  f1_t2  f2_t2
0      3      7      4      4
1      7      0      6      0
2      2      0      2      4
3      3      9      3      4
4      1      2      3      0

We will now take the values to work with them. The thing here is that RNNs incorporate the "timestep" dimension to their input, because of their architechtural nature. We can imagine that dimension as stacking 2D arrays one behind the other for the number of timesteps we have. In this case, we have two timesteps; so we will have two 2D arrays stacked: one for timestep1 and behind that, the one for timestep2.

In reality, in that 3D input we need to make, we still have 5 observations. The thing is that we need to arrange them differently: the RNN will take the first row (or specified batch - but we will keep it simple here) of the first array (i.e. timestep1) and the first row of the second stacked array (i.e. timestep2). Then the second row...until the last one (the 5th one in our example). So, in each row of each timestep, we need to have the two features, of course, separated in different arrays each one corresponding to its timestep. Let's see this with the numbers.

I will make two arrays for easier understanding. Remember that, because of our organizational scheme in the df, you might have noticed that we need to take the first two columns (i.e. features 1 and 2 for the timestep1) as our FIRST ARRAY OF THE STACK and the last two columns, that is, the 3rd and the 4th, as our SECOND ARRAY OF THE STACK, so that everything makes sense finally.

In [7]: arrStack1 = arr[:,0:2]                                                       

In [8]: arrStack1                                                                    
Out[8]: 
array([[3, 7],
       [7, 0],
       [2, 0],
       [3, 9],
       [1, 2]])

In [9]: arrStack2 = arr[:,2:4]                                                       

In [10]: arrStack2                                                                   
Out[10]: 
array([[4, 4],
       [6, 0],
       [2, 4],
       [3, 4],
       [3, 0]])

Finally, the only thing we need to do is stack both arrays ("one behind the other") as if they were part of the same final structure:

In [11]: arrfinal3D = np.stack([arrStack1, arrStack2])                               

In [12]: arrfinal3D                                                                  
Out[12]: 
array([[[3, 7],
        [7, 0],
        [2, 0],
        [3, 9],
        [1, 2]],

       [[4, 4],
        [6, 0],
        [2, 4],
        [3, 4],
        [3, 0]]])

In [13]: arrfinal3D.shape                                                            
Out[13]: (2, 5, 2)

That's it: we have our feature matrix ready to be fed into the RNN cell, taking into account our organization of the 2D feature matrix.

(For a one liner regarding all this you could use:

In [14]: arrfinal3D_1 = np.stack([arr[:,0:2], arr[:,2:4]])                           

In [15]: arrfinal3D_1                                                                
Out[15]: 
array([[[3, 7],
        [7, 0],
        [2, 0],
        [3, 9],
        [1, 2]],

       [[4, 4],
        [6, 0],
        [2, 4],
        [3, 4],
        [3, 0]]])
6
  • shouldn't the final 3d shape be (5,2,2),,5 for sample size, 2 for 2 timeseteps and 2 for features. Also shouldn't each stacked 2d layer corresponds to different features ? Commented Dec 19, 2019 at 6:51
  • No, that's not how numpy represents 3d shapes. It is common to misunderstand it, but you can try with other examples of 3d shapes to see that it is the way highlighted in the post. So, it is not (x, y, z); instead, it is (z, x, y). Regarding your second question, it is also incorrect; timesteps are what are stacked but the features are the same in each timestep.
    – Ezarate11
    Commented Dec 19, 2019 at 10:09
  • But isn't the input_shape defined as (sample_size,timestep, features). ? That's tensorflow site mentions about input_shape. Commented Dec 19, 2019 at 13:15
  • One thing is how numpy represents the shapes and the other is how you need to feed them to tensorflow. This being the case, you will need to reshape your data to fit each of the different input_shapes. This was just an exercise to understand the data dimensions, outside any package.
    – Ezarate11
    Commented Dec 19, 2019 at 17:12
  • I was just trying with numpy using reshape. So numpy represent( sample size, timestep, features) as ( features, sample size , timesteps) . Am i right ? Commented Dec 19, 2019 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.