0

I'm training a 3D CNN (U-Net) with TensorFlow 2 and Keras. I'm pretty perplexed at the loss curves I'm getting:

Loss function legend: bs=batch_size, eager=executed in eager mode

The model starts learning very well, and then in the middle of the first epoch it suddenly stops and only returns empty segmentations (the predicted output of the model). I checked by saving images/predictions to disk as the model was training. From ~15k on (red curve, ~5k green), only blank outputs (all zeros, consistently). The images and true segmentations (masks) seem just fine, even after that shift in model behavior.

The loss function is weighted cross entropy, which I checked separately:

def weighted_cross_entropy(weight_alpha=0.9):
    def _loss(y_true, y_pred):
        y_true = K.cast(y_true, y_pred.dtype)
        weights = y_true * (weight_alpha/(1.-weight_alpha)) + 1.
        bce = K.binary_crossentropy(y_true, y_pred, from_logits=False)
        weighted_loss = K.mean(bce * weights)
        return weighted_loss
    return _loss

The optimizer is tf.keras.optimizers.Adam(learning_rate=1e-4).

The dataset comes from tfrecords and is a tf.Dataset:

ds = tf.data.TFRecordDataset(tfrecords)
ds = ds.map(load_image_train, num_parallel_calls=tf.data.experimental.AUTOTUNE)
ds = ds.shuffle(100)
ds = ds.batch(batch_size)

Any thoughts?

1 Answer 1

0

There might be millions of possibilities accounting for your model performing eerie things.

You could first try decreasing the learning rate to a very small number.

There might be implementation mistakes if your model isn't trained with tensorflow functions(ex model.fit) because these functions keep on doing the same thing for at least one epoch.

This question is unanswerable without the full code give, because your fractured part seems to have no problem. Please post the training loop and model architecture together.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.