33

I need a conditional control flow in my graph. If pred is True, the graph should call an op that updates a variable and then returns it, otherwise it returns the variable unchanged. A simplified version is:

pred = tf.constant(True)
x = tf.Variable([1])
assign_x_2 = tf.assign(x, [2])
def update_x_2():
  with tf.control_dependencies([assign_x_2]):
    return tf.identity(x)
y = tf.cond(pred, update_x_2, lambda: tf.identity(x))
with tf.Session() as session:
  session.run(tf.initialize_all_variables())
  print(y.eval())

However, I find that both pred=True and pred=False lead to the same result y=[2], which means the assign op is also called when update_x_2 is not selected by tf.cond. How to explain this? And how to solve this problem?

2 Answers 2

42

TL;DR: If you want tf.cond() to perform a side effect (like an assignment) in one of the branches, you must create the op that performs the side effect inside the function that you pass to tf.cond().

The behavior of tf.cond() is a little unintuitive. Because execution in a TensorFlow graph flows forward through the graph, all operations that you refer to in either branch must execute before the conditional is evaluated. This means that both the true and the false branches receive a control dependency on the tf.assign() op, and so y always gets set to 2, even if pred is False.

The solution is to create the tf.assign() op inside the function that defines the true branch. For example, you could structure your code as follows:

pred = tf.placeholder(tf.bool, shape=[])
x = tf.Variable([1])
def update_x_2():
  with tf.control_dependencies([tf.assign(x, [2])]):
    return tf.identity(x)
y = tf.cond(pred, update_x_2, lambda: tf.identity(x))
with tf.Session() as session:
  session.run(tf.initialize_all_variables())
  print(y.eval(feed_dict={pred: False}))  # ==> [1]
  print(y.eval(feed_dict={pred: True}))   # ==> [2]
4
  • 1
    Yeah, that's the one that confuses me also. My understand is that before executing tf.cond, the runtime makes sure all the dependencies are executed. Dependencies of ops in True and False branches are also dependencies of cond, so even though an op in a branch may never be executed, all of it's dependencies are executed, does that that sound right? May 6, 2016 at 18:12
  • 2
    Yep - the graph pruning considers all potential dependencies (of either branch) for execution, and only inhibits their execution if they were defined inside one of the branches, because the CondContext adds a control dependency on the pivot and that dependency will be a dead tensor (preventing the op from executing) if it is in the branch not taken.
    – mrry
    May 6, 2016 at 18:20
  • 2
    What was the reasoning doing it this way? Why not prune the subgraph behind the non-active branch?
    – Lenar Hoyt
    Jul 1, 2017 at 16:15
  • 1
    @LenarHoyt: The pruning happens before the value for pred has been computed. This enables TensorFlow to cache a single pruned graph based on a simple key (essentially the arguments to Session.run()), and makes the implementation of conditional execution simple and lightweight. The same mechanism is used to implement tf.while_loop(), where the advantages of performing the control flow at this level are more evident.
    – mrry
    Jul 5, 2017 at 23:27
3
pred = tf.constant(False)
x = tf.Variable([1])

def update_x_2():
    assign_x_2 = tf.assign(x, [2])
    with tf.control_dependencies([assign_x_2]):
        return tf.identity(x)
y = tf.cond(pred, update_x_2, lambda: tf.identity(x))
with tf.Session() as session:
  session.run(tf.initialize_all_variables())
  print(y.eval())

This will get the result of [1].

This answer is quite the same as the above answer. But what I wanna share is you can put every ops you would like to use in its branch function. Because, given your example code, tensor x is can be directly used by the update_x_2 function.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.