13

Just like this:

x = keras.layers.Input(shape=(3,))
y = keras.layers.Dense(5)(x)

G = keras.models.Model(x, y,name='G')
G.compile(optimizer='rmsprop', loss='mse')

data_x = np.random.random((10, 3))
data_y = np.random.random((10, 5))

G.fit(data_x,data_y,shuffle=False,validation_data=[data_x,data_y],verbose=1)

Result:

Train on 10 samples, validate on 10 samples
Epoch 1/1
10/10 [==============================] - 27s 3s/step - loss: 0.4482 - val_loss: 0.4389

The printed loss and val_loss are different.In some other test,I found the difference is significant. Why?

1

4 Answers 4

9

There are some additional reasons that might have caused the observed difference in the values:

  1. According to the answer to this question of mine, the displayed training loss is computed before the optimization. So also in the case when you only train on a single batch, there is still an optimization step applied between training and validation loss evaluation.

  2. There are layers that behave differently in training phase / testing phase, for example BatchNormalization layers or Dropout layers, as explained in the Keras FAQ. If you follow the link, there is also a code example how to get the model output for either of the two phases (without applying the optimization that is applied when you call methods like model.fit, model.train_on_batch etc.)

  3. This is for completeness, although the differences would be way smaller than the ones that you have shown. When using GPU, there are some methods that may be executed non-deterministically. This may show in slight numerical differences when executing the same operation several times, although I am not sure whether it would be an issue in your concrete computation. Refer for example to the answers to this question that regards Tensorflow, or this comment that regards Theano.

1
  • 1
    Yes, the BN layer results in the significant difference between trainning loss and val loss.
    – spider
    Mar 14, 2018 at 7:20
1

Loss is a number which is calculated 'on the fly' while training the epoch. Therefore, is not really accurate. Val_loss on the other hand is calculated at the end of the epoch. Sometimes you can see this behavior in a jumping loss value from the end of the last epoch to the beginning of the next one. So the behavior is not nice, but normal.

If you want a real loss vs. val_loss comparison you should write a custom callback and calculate it yourself.

0

The training loss that you see in the progress bar is the average of loss over training batches. As the model is constantly adapting and changing during training, this number is just an indicator and not a real loss value.

The validation loss is computed at the end of an epoch, while the model is constant. That is the primary different because both values are not the same, even if you use the same data.

4
  • If training with whole data size and only one epoch, the average training loss is the same as that of one batch, val loss is computed at the end of an epoch, so, it should be the same as training loss? But in the above test, it's not.
    – spider
    Mar 8, 2018 at 10:16
  • @spider You mean with batch size equal to the whole dataset?
    – Dr. Snoopy
    Mar 8, 2018 at 10:17
  • @ Matias Valdenegro Yes. I find the problem is that the training loss is computed before net optimization, but validation loss is after net optimization.
    – spider
    Mar 8, 2018 at 10:33
  • @MatiasValdenegro What If you set validation_split parameter? Than a portion of data is excluded from training and only used for validation (the same validation data in every epoch) right? Mar 8, 2018 at 11:01
0

The difference is that the validation loss is calculated after the gradient descent on the whole epoch and the training loss is calculated before the gradient descent on this particular example. In case you actually converged both losses should be the same. You can test this by setting your learning rate to something ridicolously small like 1E-10 and checking if the losses are similar (e.g. difference being < 10E-6).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.