Supoose I have two models foo
and bar
. Assume bar
is pretrained and loaded. I want to define the cost function for foo
as roughly sketched in the following code (it's actually an autoencoder). Please note that this is an minimal example to reproduce my problem so they don't make sense mathematically.
import tensorflow as tf
def foo(X):
with tf.variable_scope("foo"):
A = tf.get_variable("A",shape=[1])
return tf.add(X,A)
def bar(X):
with tf.variable_scope("bar"):
B = tf.get_variable("B",shape=[1])
return tf.multiply(X,B)
X = tf.placeholder("float")
X_prime = foo(X)
Y = bar(X)
tf.get_variable_scope().reuse_variables()
Y_prime = bar(X_prime)
#foo(X) is manipulated with some other terms, but the point is foo is called again
cost = foo(X) + tf.pow(Y-Y_prime,2)
optimizer = tf.train.AdamOptimizer(learning_rate=0.01).minimize(cost)
If I run the script (TF version 1.0), I get the following error:
ValueError: Variable foo/A/Adam/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?
However, this doesn't happen with GradientDescentOptimizer
. Any explanation and pointer would be appreciated.
tf.get_variable_scope().reuse_variables()
, all variables after that line will look for the existing variables. If there is no such variable, tensorflow will prompt that error. Optimizers like Momentum, Adam, they need to create variables to store the historical gradients when they try tominimize
the cost. A possible way to solve this is that you can setvariable_scope("foo", reuse=reuse)
locally by adding a parameter for the functions, not globally. – LI Xuhong Mar 31 '17 at 13:21