11

According to the PyTorch documentation, the advantage of the class BCEWithLogitsLoss() is that one can use the

log-sum-exp trick for numerical stability.

If we use the class BCEWithLogitsLoss() with the parameter reduction set to None, they have a formula for that:

Loss

I now simplified the terms, and obtain after some lines of calculation:

enter image description here

I was curious to see whether this is the way how the Source code does it, but I couldn't find it.. The only code they have is this:

Code for BCEWithLogitsLoss

3 Answers 3

16

nn.BCEWithLogitsLoss is actually just cross entropy loss that comes inside a sigmoid function. It may be used in case your model's output layer is not wrapped with sigmoid. Typically used with the raw output of a single output layer neuron.

Simply put, your model's output say pred will be a raw value. In order to get probability, you will have to use torch.sigmoid(pred). (To get actual class labels, you need torch.round(torch.sigmoid(pred)).) However, you don't need to do anything like that (i.e take sigmoid) when you use nn.BCEWithLogitsLoss. Here you just have to do the following-

criterion = nn.BCEWithLogitsLoss()
loss = criterion(pred, target) # pred is just raw nn output

Hence, coming to implementation part, criterion accepts two torch tensors - one being the raw nn outputs, the other being the true class labels, then wraps the first using sigmoid - for each element in the tensor and then calculates Cross Entropy loss (-(target*log(sigmoid(pred))) for each pair and reduces it to mean.

11

All the pytorch functional code is implemented in C++. The source code for the implementation is located here.

The pytorch implementation computes BCEWithLogitsLoss as

enter image description here

where t_n is simply -relu(x). The use of t_n here is basically a clever way to avoid taking exponentials of positive values (thus avoiding overflow). This can be made more clear by substituting t_n into the l_n which yields the following equivalent expression

enter image description here

0

According to C++ implementation they use this function in the end:

static inline at::Tensor apply_loss_reduction(const at::Tensor& unreduced, int64_t reduction) {
    if (reduction == at::Reduction::Mean) {
      return unreduced.mean();
    } else if (reduction == at::Reduction::Sum) {
      return unreduced.sum();
    }
    return unreduced;
  }

As you can see in documentation they use mean reduction by default.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.