0

I'm implementing an encoder-decoder rnn by using tf.contrib.cudnn_rnn.CudnnGRU() as the encoder and I've found a problem:

I want to reuse the variables so I can create the same model but use it with other data and to put it simple this would be the code to reproduce my problem:

tf.reset_default_graph()

def create_model():
    return tf.contrib.cudnn_rnn.CudnnGRU(num_layers=1, num_units=100,
                         direction='unidirectional')

# (time, batch_size, num_inputs)
x = tf.random_normal((100, 16, 100))

with tf.variable_scope('model') as scope:
    model_1 = create_model()
    rnn_out_1, rnn_state_1 = model_1(x)
    scope.reuse_variables()
    model_2 = create_model()
    rnn_out_2, rnn_state_2 = model_2(x)

This throws the following error:

Variable model/cudnn_gru_1/opaque_kernel does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?

So the second model is trying to find the model/cudnn_gru_1/opaque_kernel variable but can not find it because it should be looking for model/cudnn_gru/opaque_kernel:0.

The thing is I don't know why this is happening since by following the Variables tensorflow reference it seems to be ok. On the other hand, I also tried to write it differently as tensorflow doc states that my above implementation and the one I'm showing next do actually the same:

tf.reset_default_graph()

def create_model():
    return tf.contrib.cudnn_rnn.CudnnGRU(num_layers=1, num_units=100,
                         direction='unidirectional')

# (time, batch_size, num_inputs)
x = tf.random_normal((100, 16, 100))

with tf.variable_scope('model'):
    model_1 = create_model()
    rnn_out_1, rnn_state_1 = model_1(x)
with tf.variable_scope('model', reuse=True):
    model_2 = create_model()
    rnn_out_2, rnn_state_2 = model_2(x)

This second way is actually working (or at least I think it is). So I don't really know what I'm doing wrong in the first implementation, I'm also not sure on wether both implementations should be doing the same (which I think they should). So does anyone please can help me figure out what I'm doing wrong or the things that I'm not able to understand properly?

Thanks in advance

1 Answer 1

2

CudnnGRU looks like keras-style Model object. So you should reuse the object to share parameters among layers like

def create_model():
    return tf.contrib.cudnn_rnn.CudnnGRU(num_layers=1, num_units=100,
                                  direction='unidirectional')


# (time, batch_size, num_inputs)
x = tf.random_normal((100, 16, 100))

model = create_model()
rnn_out_1, rnn_state_1 = model(x)
rnn_out_2, rnn_state_2 = model(x)

I don't know why only the second way is running correctly.

EDIT

I found CudnnGRU makes variable names for its variables uniquely in its current variable scope.

In the first way model_2 makes an new name like cudnn_gru_1 to make its name unique. On the other hand, in the second way you made a new variable scope, so the unique variable names of model_2 matches those of model_1.

You can found why CudnnGRU makes unique variable name in Layer._set_scope() (tensorflow\python\layers\base.py#L150). Layer class makes a new variable scope for its variable with default_name argument (scope is None in this case), so its name become unique.

2
  • So this is actually not a problem with CudnnGRU but with every layer? And I understand what you say about the two different ways and their differences, but shouldn't they actually have if well implemented the same behaviour? Should it be reported as a bug?
    – ERed
    Commented Oct 21, 2018 at 13:30
  • Yes I expect this is a common problem among layers inheriting from Layer. This behavior is not intuitive, but we should use the same object of Layer for parameter sharing, so I think this is not a important bug.
    – saket
    Commented Oct 21, 2018 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.