I'm testing the tf.variable_scope object to reuse a network multiple times within same session.run call:
data1 = tf.constant([[[3.,5.,6.,1.]]],dtype=tf.float64)
data2 = tf.constant(np.zeros((1,5)))
def networkS(input_1, input_2, reuse):
#this is a multi-input network using tf.keras api
with tf.variable_scope("test", reuse=reuse):
#input_1
x = tf.keras.layers.CuDNNGRU(512, return_sequences=True)(input_1)
x = tf.keras.layers.CuDNNGRU(512, return_sequences=True)(x)
x = tf.keras.layers.CuDNNGRU(512)(x)
#input_2
y = tf.keras.layers.Dense(32, activation="relu")(input_2)
#merge two input
x = tf.keras.layers.concatenate([x, y], axis=-1)
x = tf.keras.layers.Dense(512, activation='relu')(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dense(128, activation='relu')(x)
vf_pre = tf.keras.layers.Dense(128, activation='relu')(x)
vf = tf.keras.layers.Dense(1)(vf_pre)
return vf
v1 = networkS(data1, data2, tf.AUTO_REUSE)
v2 = networkS(data1, data2, tf.AUTO_REUSE)
v3 = networkS(data1, data2, tf.AUTO_REUSE)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run([v1,v2,v3]))
My understanding is that, during the graph construction phase: the first call to networkS(), we create a new network under variavle_scope "test", then the following calls to networkS() just reuse the existing layer variable. However, with identical inputs, we have different results for v1, v2 ,v3
[array([[0.00112361]]), array([[0.00107469]]), array([[0.00115032]])]
I think this means the three network are constructed in paralell and fail to share the same variables, thus producing different results from same inputs.
If I call sess.run twice, it does produce same results between calls
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run([v1,v2,v3]))
print(sess.run([v1,v2,v3]))
[array([[0.00550815]]), array([[-0.00294633]]), array([[0.00584344]])]
[array([[0.00550815]]), array([[-0.00294633]]), array([[0.00584344]])]
How do I solve this problem?
v1 = networkS(data1, data2, False) v2 = networkS(data1, data2, True) v3 = networkS(data1, data2, True)
, same results – Gil Jan 12 at 18:22