6

I am attempting to use a batch norm layer from TensorFlow-Slim as follows:

net = ...
net = slim.batch_norm(net, scale = True, is_training = self.isTraining,
    updates_collections = None, decay = 0.9)
net = tf.nn.relu(net)
net = ...

I train with:

self.optimizer = slim.learning.create_train_op(self.model.loss,
    tf.train.MomentumOptimizer(learning_rate = self.learningRate,
    momentum = 0.9, use_nesterov = True)

optimizer = self.sess.run([self.optimizer],
    feed_dict={self.model.isTraining:True})

I load the saved weights with:

net = model.Model(sess,width,height,channels,weightDecay)

savedWeightsDir = './savedWeights/'
saver = tf.train.Saver(max_to_keep = 5)
checkpointStr = tf.train.latest_checkpoint(savedWeightsDir)
sess.run(tf.global_variables_initializer())
saver.restore(sess, checkpointStr)
global_step = tf.contrib.framework.get_or_create_global_step()

and I infer with:

inf = self.sess.run([self.softmax],
    feed_dict = {self.imageBatch:imageBatch,self.isTraining:False})

Of course I left out a lot and paraphrased some code, but I think this is all that the batch norm touches. The odd thing is, if I set isTraining:True, I get much better results. Could it be something with loading in the weights - perhaps the batch norm values are not saved? Is there anything obviously wrong in the code? Thank you.

1 Answer 1

0

I've just run into the same issue and found solution here. The problem originates from tf.layers.batch_normalization layer which has the need to update moving_mean and moving_variance.

In order to do that correctly in your case you need to modify your training process as:

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    self.optimizer = slim.learning.create_train_op(self.model.loss,
      tf.train.MomentumOptimizer(learning_rate = self.learningRate,
      momentum = 0.9, use_nesterov = True)

or more generally, from documentation:

  x_norm = tf.layers.batch_normalization(x, training=training)

  # ...

  update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
  with tf.control_dependencies(update_ops):
    train_op = optimizer.minimize(loss)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.