16

How to change the learning rate of Adam optimizer, while learning is progressing in TF2? There some answers floating around, but applicable to TF1, e.g. using feed_dict.

4 Answers 4

20

If you are using custom training loop (instead of keras.fit()), you can simply do:

new_learning_rate = 0.01 
my_optimizer.lr.assign(new_learning_rate)
1
18

You can read and assign the learning rate via a callback. So you can use something like this:

class LearningRateReducerCb(tf.keras.callbacks.Callback):

  def on_epoch_end(self, epoch, logs={}):
    old_lr = self.model.optimizer.lr.read_value()
    new_lr = old_lr * 0.99
    print("\nEpoch: {}. Reducing Learning Rate from {} to {}".format(epoch, old_lr, new_lr))
    self.model.optimizer.lr.assign(new_lr)

Which, for example, using the MNIST demo can be applied like this:

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, callbacks=[LearningRateReducerCb()], epochs=5)

model.evaluate(x_test, y_test)

giving output like this:

Train on 60000 samples
Epoch 1/5
59744/60000 [============================>.] - ETA: 0s - loss: 0.2969 - accuracy: 0.9151
Epoch: 0. Reducing Learning Rate from 0.0010000000474974513 to 0.0009900000877678394
60000/60000 [==============================] - 6s 92us/sample - loss: 0.2965 - accuracy: 0.9152
Epoch 2/5
59488/60000 [============================>.] - ETA: 0s - loss: 0.1421 - accuracy: 0.9585
Epoch: 1. Reducing Learning Rate from 0.0009900000877678394 to 0.000980100128799677
60000/60000 [==============================] - 5s 91us/sample - loss: 0.1420 - accuracy: 0.9586
Epoch 3/5
59968/60000 [============================>.] - ETA: 0s - loss: 0.1056 - accuracy: 0.9684
Epoch: 2. Reducing Learning Rate from 0.000980100128799677 to 0.0009702991228550673
60000/60000 [==============================] - 5s 91us/sample - loss: 0.1056 - accuracy: 0.9684
Epoch 4/5
59520/60000 [============================>.] - ETA: 0s - loss: 0.0856 - accuracy: 0.9734
Epoch: 3. Reducing Learning Rate from 0.0009702991228550673 to 0.0009605961386114359
60000/60000 [==============================] - 5s 89us/sample - loss: 0.0857 - accuracy: 0.9733
Epoch 5/5
59712/60000 [============================>.] - ETA: 0s - loss: 0.0734 - accuracy: 0.9772
Epoch: 4. Reducing Learning Rate from 0.0009605961386114359 to 0.0009509901865385473
60000/60000 [==============================] - 5s 87us/sample - loss: 0.0733 - accuracy: 0.9772
10000/10000 [==============================] - 0s 43us/sample - loss: 0.0768 - accuracy: 0.9762
[0.07680597708942369, 0.9762]
2
  • Thank you. It appears that I do not need even callback, I just need to execute optimizer.lr.assign(new_value) Commented Aug 2, 2019 at 1:27
  • My pleasure. Glad it helps. If my answer was helpful to you then you can always upvote and/or accept.. :-)
    – Stewart_R
    Commented Aug 2, 2019 at 5:06
9

If you want to use low-level control and not the fit functionality with callbacks, have a look at tf.optimizers.schedules. Here's some example code:

train_steps = 25000
lr_fn = tf.optimizers.schedules.PolynomialDecay(1e-3, train_steps, 1e-5, 2)
opt = tf.optimizers.Adam(lr_fn)

This would decay the learning rate from 1e-3 to 1e-5 over 25000 steps with a power-2 polynomial decay.

Note:

  • This doesn't really "store" a learning rate as in the other answer, but rather the learning rate is now a function that will be called every time it is needed to compute the current learning rate.
  • Optimizer instances have an internal step counter that will count up by one each time apply_gradients is called (as far as I can tell...). This allows for this procedure to work properly when using it in a low-level context (usually with tf.GradientTape)
  • Unfortunately this feature is not well-documented (docs just say that the learning rate argument has to be a float or tensor...) but it works. You can also write your own decay schedules. I think they just need to be functions that take in some current "state" of the optimizer (probably number of training steps) and return a float to be used as learning rate.
1
  • 2
    This is really hard to find but the most elegant solution imho assigning it during the loop is very ugly
    – Fabricio
    Commented Mar 25, 2020 at 18:27
3

You have 3 solutions:

Here is an example from this tutorial:

class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
    def __init__(self, d_model, warmup_steps=4000):
        super(CustomSchedule, self).__init__()

        self.d_model = d_model
        self.d_model = tf.cast(self.d_model, tf.float32)

        self.warmup_steps = warmup_steps

    def __call__(self, step):
        arg1 = tf.math.rsqrt(step)
        arg2 = step * (self.warmup_steps ** -1.5)

        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)

And you pass it to your optimizer:

learning_rate = CustomSchedule(d_model)

optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, 
                                     epsilon=1e-9)

This way, the CustomSchedule will be part of your graph and it will update the Learning rate while your model is training.

3
  • What is d_model?
    – Nerxis
    Commented Apr 28, 2021 at 14:55
  • @Nerxis The model where you have set you're optimizer on
    – Begoodpy
    Commented Apr 30, 2021 at 8:05
  • Thanks, I just found this in the example, I think it would be worth of adding this directly to your answer. You just copied content of the example but without whole context it's not clear.
    – Nerxis
    Commented Apr 30, 2021 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.