The documentation is not quite clear about this. I suppose the gradients one can obtain by opt.compute_gradients(E, [v])
contain the ∂E/∂x = g(x)
for each element x
of the tensor that v
stores. Does opt.apply_gradients(grads_and_vars)
essentially execute x ← -η·g(x)
, where η
is the learning rate? That would imply that if I want to add a positive additive change p
to the variable, I would need to need to change g(x) ← g(x) - (1/η)p
, e.g. like this:
opt = tf.train.GradientDescentOptimizer(learning_rate=l)
grads_and_vars = opt.compute_gradients(loss, var_list)
for l, gv in enumerate(grads_and_vars):
grads_and_vars[l] = (gv[0] - (1/l) * p, gv[1])
train_op = opt.apply_gradients(grads_and_vars)
Is there a better way to do this?