You can use the global_step
variable to keep track of steps, but if in your code, you are initializing or assigning this value to another step
variable, it may not be consistent.
For instance, you define your global_step
using:
global_step = tf.Variable(0, name='global_step', trainable=False)
Assign to your training operation:
train_op = optimizer.minimize(loss, global_step=global_step)
Save in your checkpoint:
saver.save(sess, checkpoint_path, global_step=global_step)
And restore from your checkpoint:
saver.restore(sess, checkpoint_path)
the value of global_step
is restored as well but if you are assigning it to another variable, say step
, then you must do something like:
step = global_step.eval(session=sess)
The variable step
, contains the last saved global_step
in the checkpoint.
It will be nice to also define the global_step from graph than as zero variable (as earlier defined):
global_step = tf.train.get_or_create_global_step()
This will get your last global_step
if exist or create one if not.