4

I see a lot of explanations about CEL or binary cross entropy loss in the context where the ground truth is say, a 0 or 1, and then you get a function like:

def CrossEntropy(yHat, y):
    if yHat == 1:
      return -log(y)
    else:
      return -log(1 - y)

However, I'm confused at how BCE works when your yHat is not a discrete 0 or 1. For example if I want to look at reconstruction loss for an MNIST digit where my ground truths are 0 < yHat < 1, and my predictions are also in the same range, how does this change my function?

EDIT:

Apologies let me give some more context for my confusion. In the PyTorch tutorials on VAEs they use BCE to calculate reconstruction loss, where yhat (as far as I understand, is not discrete). See:

https://github.com/pytorch/examples/blob/master/vae/main.py

The implementation works...but I don't understand how that BCE loss is calculated in this case.

6
  • For autoencoders with images you can normalize your pixel values to range [0, 1] and then use BCE pixel-wise
    – Coolness
    Mar 25, 2018 at 20:27
  • Sure, but is that what they're doing here?
    – Matt
    Mar 25, 2018 at 20:29
  • In the code that you posted, the code handles more than just 0 and 1. The first if statement handles the case of 1, but the else statement handles all other values, not just 0. Mar 25, 2018 at 20:50
  • 1
    Check out pytorch.org/docs/master/… and pytorch.org/docs/master/nn.html#torch.nn.BCEWithLogitsLoss they take the sigmoid function from the data, so it gets normalized into [0, 1]
    – Coolness
    Mar 25, 2018 at 20:54
  • @stackoverflowuser2010 - right, but this code would not work properly (as in calculate the proper CE loss) if it took anything that wasn't 0 or 1.
    – Matt
    Mar 25, 2018 at 20:58

3 Answers 3

5

Cross entropy measures distance between any two probability distributions. In what you describe (the VAE), MNIST image pixels are interpreted as probabilities for pixels being "on/off". In that case your target probability distribution is simply not a dirac distribution (0 or 1) but can have different values. See the cross entropy definition on Wikipedia.

With the above as a reference, let's say your model outputs a reconstruction for a certain pixel of 0.7. This essentially says that your model estimates p(pixel=1) = 0.7, and accordingly p(pixel=0) = 0.3.
If the target pixels would just be 0 or 1, your cross entropy for this pixel would either be -log(0.3) if the true pixel is 0 or -log(0.7) (a smaller value) if the true pixel is 1.
The full formula would be -(0*log(0.3) + 1*log(0.7)) if the true pixel is 1 or -(1*log(0.3) + 1*log(0.7)) otherwise.

Let's say your target pixel is actually 0.6! This essentially says that the pixel has a probability of 0.6 to be on and 0.4 to be off.
This simply changes the cross entropy computation to -(0.4*log(0.3) + 0.6*log(0.7)).

Finally, you can simply average/sum these per-pixel cross-entropies over the image.

0

The cross-entropy loss is only used in classification problems: i.e., where your target (yHat) is discrete. If you have a regression problem instead, something like the mean squared error (MSE) loss would be more appropriate. You can find a variety of losses for the PyTorch library, and their implementations here.

In the case of the MNIST dataset, you actually have a multiclass classification problem (you're trying to predict the correct digit out of 10 possible digits), so the binary cross-entropy loss isn't suitable, and you should the general cross-entropy loss instead.

Regardless, the first step in your investigation should be identifying whether your problem is "classification" or "regression". A loss function suitable for one problem is generally not suitable for the other.

EDIT: You can find a more detailed explanation of the cross-entropy loss in the context of the MNIST problem at the "MNIST for ML Beginners" tutorial on the TensorFlow website.

5
  • 1
    Hmm im mainly confused in the context of VAEs. For example, in the pytorch official example, BCE is used for reconstruction loss: github.com/pytorch/examples/blob/master/vae/main.py
    – Matt
    Mar 25, 2018 at 20:18
  • The adversarial problem is different from the traditional MNIST classification problem. In the adversarial problem, the second net tries to output a probability that the image generated by the first net is of real data. This is a binary problem: real vs. fake. In the traditional classification problem, the (single) net tries to output the probabilities that the image corresponds to any of the digits. This is a multiclass problem: 0 vs 1 vs 2 vs etc.
    – vbox
    Mar 25, 2018 at 20:20
  • Anyway, for the real vs. fake problem, BCE is appropriate. The function you've written should work as normal, except yHat is 0 if the image is fake and 1 if it is real; y is the probability of it being real (generated by the second net).
    – vbox
    Mar 25, 2018 at 20:20
  • I think you misunderstand, I'm referring to a variational autoencoder, not a GAN.
    – Matt
    Mar 25, 2018 at 20:21
  • Also, I would love to see the source but I had a bit of trouble digging for it. If I remember correctly the nn modules call the F. modules, which in turn for BCE calls something else (a file in C?), and I had trouble actually digging it up...
    – Matt
    Mar 25, 2018 at 20:26
0

You generally shouldn't encode set of classes that isn't binary as values between 0 and 1. In the case of MNIST, if you were to label each digit 0, 0.1, 0.2, etc, this implies that an image of a 2 is more similar to an image of a 0 than an image of a 5, which isn't necessarily true.

One good thing to do is to "one hot encode" your labels instead, as a 10 element array of 0s. Then, set the index corresponding to the digit image to 1.

As mentioned above, you would then use the regular cross-entropy loss function. Your model should then output a vector of conditional probabilities for each sample, corresponding to each possible class. Probably using a softmax function.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.