158

What would cause a Convolutional Neural Network to diverge?

Specifics:

I am using Tensorflow's iris_training model with some of my own data and keep getting

ERROR:tensorflow:Model diverged with loss = NaN.

Traceback...

tensorflow.contrib.learn.python.learn.monitors.NanLossDuringTrainingError: NaN loss during training.

Traceback originated with line:

 tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                        hidden_units=[300, 300, 300],
                                        #optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.001, l1_regularization_strength=0.00001),                                                          
                                        n_classes=11,
                                        model_dir="/tmp/iris_model")

I've tried adjusting the optimizer, using a zero for learning rate, and using no optimizer.

5
  • 6
    It's a natural property of stochastic gradient descent, if the learning rate is too large, SGD can diverge into infinity Commented Oct 14, 2016 at 20:02
  • @YaroslavBulatov I've tried with that AdagradOptiizer with a learning rate of about 1E-15. Perhaps my data isn't suited to SGD, can you suggest another algorithm? Still new to Tensorflow and Deep Learning.
    – Free Url
    Commented Oct 14, 2016 at 20:13
  • If you'd like to gather more information on the error and if the error occurs in the first few iterations, I suggest you run the experiment in CPU-only mode (no GPUs). The error message will be much more specific. Source: github.com/tensorflow/tensor2tensor/issues/574
    – empty
    Commented Jun 14, 2018 at 18:40
  • In my case normalization helped
    – Dmitry
    Commented Jan 9, 2019 at 19:04
  • 2
    The solution for me was using tf.losses.sparse_softmax_cross_entropy(y, logits) instead of my own implementation of Safe Softmax using tf.nn.Softmax Commented Mar 22, 2019 at 0:45

13 Answers 13

226

There are lots of things I have seen make a model diverge.

  1. Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity.

  2. I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue.

  3. Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root whose derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier.

  4. You may have an issue with the input data. Try calling assert not np.any(np.isnan(x)) on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255].

  5. The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).

9
  • 6
    thanks for the breakdown. My problem was that my labels were symmetric around zero (i.e. [-5,...,5]). Shifting solved the problem.
    – Free Url
    Commented Nov 8, 2016 at 0:45
  • 8
    The labels should be binary. 1 or 0. Otherwise the categorical cross-entropy cost function would not make sense.
    – chasep255
    Commented Nov 8, 2016 at 1:26
  • 2
    tf.keras.utils.normalize(data) was useful to normalize the data. Commented Nov 22, 2017 at 3:21
  • 2
    by 'binary' one means that they should be one-hot encoded, i.e. a vector (1,0,0,....,0) for examples of the first class, (0,1,0,....0) for examples of the second class and (0,....,0,1) for examples of the last class. The number of output nodes should be the same as the number of classes you have. Commented Jan 12, 2018 at 7:33
  • 1
    You are my hero! When I try the linear regression example (toptal.com/machine-learning/…) with another dataset, say Celsius to Fahrenheit , I got W, b, loss all 'nan'. But after follow your answer, I changed learning_rate = 0.01 to learning_rate = 0.001, then everything worked perfect!
    – holibut
    Commented Mar 16, 2018 at 8:22
24

If you're training for cross entropy, you want to add a small number like 1e-8 to your output probability.

Because log(0) is negative infinity, when your model trained enough the output distribution will be very skewed, for instance say I'm doing a 4 class output, in the beginning my probability looks like

0.25 0.25 0.25 0.25

but toward the end the probability will probably look like

1.0 0 0 0

And you take a cross entropy of this distribution everything will explode. The fix is to artifitially add a small number to all the terms to prevent this.

3
  • I use the categorical_crossentropy loss function from keras, does it already implement this? Commented Sep 10, 2018 at 3:25
  • @StayFoolish I am not sure, the cop-out answer would be to look at their source code, but I'm willing to bet they have taken-care of this in their code already. I'd try and see, most likely you're fine.
    – Evan Pu
    Commented Nov 28, 2018 at 16:20
  • 1
    And I'm assuming that 16-bit precision will face this issue much more than 32 bit precision? Commented Oct 13, 2021 at 14:59
11

In my case I got NAN when setting distant integer LABELs. ie:

  • Labels [0..100] the training was ok,
  • Labels [0..100] plus one additional label 8000, then I got NANs.

So, not use a very distant Label.

EDIT You can see the effect in the following simple code:

from keras.models import Sequential
from keras.layers import Dense, Activation
import numpy as np

X=np.random.random(size=(20,5))
y=np.random.randint(0,high=5, size=(20,1))

model = Sequential([
            Dense(10, input_dim=X.shape[1]),
            Activation('relu'),
            Dense(5),
            Activation('softmax')
            ])
model.compile(optimizer = "Adam", loss = "sparse_categorical_crossentropy", metrics = ["accuracy"] )

print('fit model with labels in range 0..5')
history = model.fit(X, y, epochs= 5 )

X = np.vstack( (X, np.random.random(size=(1,5))))
y = np.vstack( ( y, [[8000]]))
print('fit model with labels in range 0..5 plus 8000')
history = model.fit(X, y, epochs= 5 )

The result shows the NANs after adding the label 8000:

fit model with labels in range 0..5
Epoch 1/5
20/20 [==============================] - 0s 25ms/step - loss: 1.8345 - acc: 0.1500
Epoch 2/5
20/20 [==============================] - 0s 150us/step - loss: 1.8312 - acc: 0.1500
Epoch 3/5
20/20 [==============================] - 0s 151us/step - loss: 1.8273 - acc: 0.1500
Epoch 4/5
20/20 [==============================] - 0s 198us/step - loss: 1.8233 - acc: 0.1500
Epoch 5/5
20/20 [==============================] - 0s 151us/step - loss: 1.8192 - acc: 0.1500
fit model with labels in range 0..5 plus 8000
Epoch 1/5
21/21 [==============================] - 0s 142us/step - loss: nan - acc: 0.1429
Epoch 2/5
21/21 [==============================] - 0s 238us/step - loss: nan - acc: 0.2381
Epoch 3/5
21/21 [==============================] - 0s 191us/step - loss: nan - acc: 0.2381
Epoch 4/5
21/21 [==============================] - 0s 191us/step - loss: nan - acc: 0.2381
Epoch 5/5
21/21 [==============================] - 0s 188us/step - loss: nan - acc: 0.2381
2
  • Interesting. I would think this is dependent on your loss function. Can you please specify how you were measuring loss?
    – Free Url
    Commented Feb 4, 2019 at 2:14
  • 1
    I used, as is, the 'sparse_categorical_crossentropy'
    – Guido
    Commented Feb 4, 2019 at 8:26
5

If using integers as targets, makes sure they aren't symmetrical at 0.

I.e., don't use classes -1, 0, 1. Use instead 0, 1, 2.

4
  • 3
    Would you care to comment a little bit on the reasons why or cite a reference for completion?
    – gsimard
    Commented Apr 16, 2018 at 3:36
  • 1
    @gsimard Honestly I don't remember as I worked with this a while back.
    – Rok Povsic
    Commented Apr 16, 2018 at 7:31
  • 2
    @gsimard, this is because of reason 5 in the accepted answer. Logistic-based regression functions often use logarithms, which are only defined on non-negative numbers
    – Free Url
    Commented Apr 29, 2018 at 18:31
  • 1
    @Zroach No, in my case negative numbers were supported but the reason of it not working was specifically symmetry at 0.
    – Rok Povsic
    Commented Nov 15, 2018 at 5:58
3

Although most of the points are already discussed. But I would like to highlight again one more reason for NaN which is missing.

tf.estimator.DNNClassifier(
    hidden_units, feature_columns, model_dir=None, n_classes=2, weight_column=None,
    label_vocabulary=None, optimizer='Adagrad', activation_fn=tf.nn.relu,
    dropout=None, config=None, warm_start_from=None,
    loss_reduction=losses_utils.ReductionV2.SUM_OVER_BATCH_SIZE, batch_norm=False
)

By default activation function is "Relu". It could be possible that intermediate layer's generating a negative value and "Relu" convert it into the 0. Which gradually stops training.

I observed the "LeakyRelu" able to solve such problems.

2

The reason for nan, inf or -inf often comes from the fact that division by 0.0 in TensorFlow doesn't result in a division by zero exception. It could result in a nan, inf or -inf "value". In your training data you might have 0.0 and thus in your loss function it could happen that you perform a division by 0.0.

a = tf.constant([2., 0., -2.])
b = tf.constant([0., 0., 0.])
c = tf.constant([1., 1., 1.])
print((a / b) + c)

Output is the following tensor:

tf.Tensor([ inf  nan -inf], shape=(3,), dtype=float32)

Adding a small eplison (e.g., 1e-5) often does the trick. Additionally, since TensorFlow 2 the opteration tf.math.division_no_nan is defined.

2

Check your output layer. If you have actual classes set to 5, ensure that the output layer is configured to have 5 classes as well. If you mistakenly set it to 3 or any other value, during training you may encounter 'NaN' (not a number) errors.

OP = L.Dense(5, activation="softmax")(x)
1

Regularization can help. For a classifier, there is a good case for activity regularization, whether it is binary or a multi-class classifier. For a regressor, kernel regularization might be more appropriate.

0

I found some interesting thing when battling whit this problem,in addition to the above answers when your data labels are arranged like below applying shuffle to data may help:

y=[0,0,0,0,0,0,0....,0,0,0,1,1,1,1,1....,1,1,1,1,1,1,1,2,2,2,2,2,......,2,2,2,2,2]

from sklearn.utils import shuffle
x, y = shuffle(x, y)
0

TensorFlow uses the labels as positions in a tensor in some contexts so they have to be 0, 1, ..., L-1. Negative numbers, non-integers, etc. can instead cause the loss to be NaN.

0

The reason could also be using very small values (like 1e9). Try replacing them with:

tf.float32.min

Or (If you manually changed tf.keras.backend.floatx):

tf.float16.min
0

I'd like to plug in some (shallow) reasons I have experienced as follows:

  1. we may have updated our dictionary(for NLP tasks) but the model and the prepared data used a different one.
  2. we may have reprocessed our data(binary tf_record) but we loaded the old model. The reprocessed data may conflict with the previous one.
  3. we may should train the model from scratch but we forgot to delete the checkpoints and the model loaded the latest parameters automatically.
0

i had a loss function like this, which involves a square root.

# diff is some difference between 2 tensors
diff = diff.pow(2).sum(dim=1).sqrt()
loss = diff.mean()

for some models, the diff tensor is very small - almost 0. the entries are something like 1e-10 or lower. i only just realized that this is problematic because the gradient of sqrt(x) is 1/(2*sqrt(x)), so if x is close to 0, the gradient will be NaN. I was initially very confused why my model suddenly diverges when none of the losses "exploded" (usually model diverges because some loss value became very big which causes the gradients to become big)

in my case, a simple fix is to bound the sqrt by some epsilon, like 1e-8

diff = diff.pow(2).sum(dim=1)
# handle case when diff is very very close to 0
diff = torch.where(diff < eps, torch.tensor(eps, device=diff.device), diff)
loss = diff.sqrt()

on hindsight, i should probably have just used torch's built-in distance losses, which I'm sure already take care of problems like this internally.

hope this helps anyone who might see this!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.