458

In the following TensorFlow function, we must feed the activation of artificial neurons in the final layer. That I understand. But I don't understand why it is called logits? Isn't that a mathematical function?

loss_function = tf.nn.softmax_cross_entropy_with_logits(
     logits = last_layer,
     labels = target_output
)
2

10 Answers 10

434

Logits is an overloaded term which can mean many different things:


In Math, Logit is a function that maps probabilities ([0, 1]) to R ((-inf, inf))

enter image description here

Probability of 0.5 corresponds to a logit of 0. Negative logit correspond to probabilities less than 0.5, positive to > 0.5.

In ML, it can be

the vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function. The softmax function then generates a vector of (normalized) probabilities with one value for each possible class.

Logits also sometimes refer to the element-wise inverse of the sigmoid function.

7
  • 196
    For Tensorflow: It's a name that it is thought to imply that this Tensor is the quantity that is being mapped to probabilities by the Softmax.
    – thertweck
    Commented May 14, 2017 at 21:46
  • 2
    is this just the same as the thing that gets exponentiated before the softmax? i.e. softmax(logit) = exp(logit)/Z(logit) then logit = h_NN(x)? so logit is the same as "score"? Commented Jan 22, 2018 at 19:29
  • 14
    Personal understanding, in TensorFlow domain, logits are the values to be used as input to softmax. I came to this understanding based on this tensorflow tutorial.
    – Diansheng
    Commented Jan 26, 2018 at 8:37
  • 4
    I am not sure whether this answers the question. Maybe that is why it was never accepted. I understand what the logit function is, but it also puzzles my why Tensorflow calls these arguments logits. It is also the same designation for several of the parameters in Tensorflow's functions
    – dleal
    Commented Jan 29, 2018 at 3:01
  • 1
    Greate!Can you make a simple example? Is this right?[1, 0.5, 0.5] through normalization become [0.5, 0.25, 0.25] and then soft max become[0,] if one hot [1, 0, 0]? or just out put [1, 0, 0] cause the output should be a vector?
    – Tina Liu
    Commented Jul 11, 2019 at 14:10
225

Just adding this clarification so that anyone who scrolls down this much can at least gets it right, since there are so many wrong answers upvoted.

Diansheng's answer and JakeJ's answer get it right.
A new answer posted by Shital Shah is an even better and more complete answer.


Yes, logit as a mathematical function in statistics, but the logit used in context of neural networks is different. Statistical logit doesn't even make any sense here.


I couldn't find a formal definition anywhere, but logit basically means:

The raw predictions which come out of the last layer of the neural network.
1. This is the very tensor on which you apply the argmax function to get the predicted class.
2. This is the very tensor which you feed into the softmax function to get the probabilities for the predicted classes.


Also, from a tutorial on official tensorflow website:

Logits Layer

The final layer in our neural network is the logits layer, which will return the raw values for our predictions. We create a dense layer with 10 neurons (one for each target class 0–9), with linear activation (the default):

logits = tf.layers.dense(inputs=dropout, units=10)

If you are still confused, the situation is like this:

raw_predictions = neural_net(input_layer)
predicted_class_index_by_raw = argmax(raw_predictions)
probabilities = softmax(raw_predictions)
predicted_class_index_by_prob = argmax(probabilities)

where, predicted_class_index_by_raw and predicted_class_index_by_prob will be equal.

Another name for raw_predictions in the above code is logit.


As for the why logit... I have no idea. Sorry.
[Edit: See this answer for the historical motivations behind the term.]


Trivia

Although, if you want to, you can apply statistical logit to probabilities that come out of the softmax function.

If the probability of a certain class is p,
Then the log-odds of that class is L = logit(p).

Also, the probability of that class can be recovered as p = sigmoid(L), using the sigmoid function.

Not very useful to calculate log-odds though.

0
164

Summary

In context of deep learning the logits layer means the layer that feeds in to softmax (or other such normalization). The output of the softmax are the probabilities for the classification task and its input is logits layer. The logits layer typically produces values from -infinity to +infinity and the softmax layer transforms it to values from 0 to 1.

Historical Context

Where does this term comes from? In 1930s and 40s, several people were trying to adapt linear regression to the problem of predicting probabilities. However linear regression produces output from -infinity to +infinity while for probabilities our desired output is 0 to 1. One way to do this is by somehow mapping the probabilities 0 to 1 to -infinity to +infinity and then use linear regression as usual. One such mapping is cumulative normal distribution that was used by Chester Ittner Bliss in 1934 and he called this "probit" model, short for "probability unit". However this function is computationally expensive while lacking some of the desirable properties for multi-class classification. In 1944 Joseph Berkson used the function log(p/(1-p)) to do this mapping and called it logit, short for "logistic unit". The term logistic regression derived from this as well.

The Confusion

Unfortunately the term logits is abused in deep learning. From pure mathematical perspective logit is a function that performs above mapping. In deep learning people started calling the layer "logits layer" that feeds in to logit function. Then people started calling the output values of this layer "logit" creating the confusion with logit the function.

TensorFlow Code

Unfortunately TensorFlow code further adds in to confusion by names like tf.nn.softmax_cross_entropy_with_logits. What does logits mean here? It just means the input of the function is supposed to be the output of last neuron layer as described above. The _with_logits suffix is redundant, confusing and pointless. Functions should be named without regards to such very specific contexts because they are simply mathematical operations that can be performed on values derived from many other domains. In fact TensorFlow has another similar function sparse_softmax_cross_entropy where they fortunately forgot to add _with_logits suffix creating inconsistency and adding in to confusion. PyTorch on the other hand simply names its function without these kind of suffixes.

Reference

The Logit/Probit lecture slides is one of the best resource to understand logit. I have also updated Wikipedia article with some of above information.

2
  • 3
    "From pure mathematical perspective logit is a function that performs above mapping." This section is wrong. It's common in statistics to call the logit of a probability itself the "logits". that feeds in to logit function the SoftMax function isn't the logit function, but its inverse, the (multinomial) logistic function.
    – iacob
    Commented Mar 25, 2021 at 19:40
  • I know I'm a "little" late to the party, but I believe there is a reason for the _with_logits suffix. A typical formula for CE loss found in papers is expressed using probabilities p (i.e. the output of softmax). However, the CE loss can be calculated using the logits (taking previous layer outputs and not doing softmax). While you rarely seen this formula written out, using logits instead of probabilities results in a more numerically stable calculation of the CE loss, and the suffix indicates that this formula is used in the implementation.
    – penelope
    Commented Jun 7 at 18:15
90

Logit is a function that maps probabilities [0, 1] to [-inf, +inf].

Softmax is a function that maps [-inf, +inf] to [0, 1] similar as Sigmoid. But Softmax also normalizes the sum of the values(output vector) to be 1.

Tensorflow "with logit": It means that you are applying a softmax function to logit numbers to normalize it. The input_vector/logit is not normalized and can scale from [-inf, inf].

This normalization is used for multiclass classification problems. And for multilabel classification problems sigmoid normalization is used i.e. tf.nn.sigmoid_cross_entropy_with_logits

2
  • 13
    so logit is the same as the "score" Commented Jan 22, 2018 at 19:29
  • 1
    I suggest adding a line in your answer explicitly differentiating Logit function (statistics) and logits layer (tensorflow) Commented May 24, 2018 at 14:40
46

Personal understanding, in TensorFlow domain, logits are the values to be used as input to softmax. I came to this understanding based on this tensorflow tutorial.

https://www.tensorflow.org/tutorials/layers


Although it is true that logit is a function in maths(especially in statistics), I don't think that's the same 'logit' you are looking at. In the book Deep Learning by Ian Goodfellow, he mentioned,

The function σ−1(x) is called the logit in statistics, but this term is more rarely used in machine learning. σ−1(x) stands for the inverse function of logistic sigmoid function.

In TensorFlow, it is frequently seen as the name of last layer. In Chapter 10 of the book Hands-on Machine Learning with Scikit-learn and TensorFLow by Aurélien Géron, I came across this paragraph, which stated logits layer clearly.

note that logits is the output of the neural network before going through the softmax activation function: for optimization reasons, we will handle the softmax computation later.

That is to say, although we use softmax as the activation function in the last layer in our design, for ease of computation, we take out logits separately. This is because it is more efficient to calculate softmax and cross-entropy loss together. Remember that cross-entropy is a cost function, not used in forward propagation.

33

(FOMOsapiens).

If you check math Logit function, it converts real space from [0,1] interval to infinity [-inf, inf].

Sigmoid and softmax will do exactly the opposite thing. They will convert the [-inf, inf] real space to [0, 1] real space.

This is why, in machine learning we may use logit before sigmoid and softmax function (since they match).

And this is why "we may call" anything in machine learning that goes in front of sigmoid or softmax function the logit.

Here is G. Hinton video using this term.

26

Here is a concise answer for future readers. Tensorflow's logit is defined as the output of a neuron without applying activation function:

logit = w*x + b,

x: input, w: weight, b: bias. That's it.


The following is irrelevant to this question.

For historical lectures, read other answers. Hats off to Tensorflow's "creatively" confusing naming convention. In PyTorch, there is only one CrossEntropyLoss and it accepts un-activated outputs. Convolutions, matrix multiplications and activations are same level operations. The design is much more modular and less confusing. This is one of the reasons why I switched from Tensorflow to PyTorch.

1
  • This might be correct but completely misses any reference/source of the information.
    – user12582392
    Commented Jun 13, 2023 at 10:42
16

logits

The vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function. The softmax function then generates a vector of (normalized) probabilities with one value for each possible class.

In addition, logits sometimes refer to the element-wise inverse of the sigmoid function. For more information, see tf.nn.sigmoid_cross_entropy_with_logits.

official tensorflow documentation

8

They are basically the fullest learned model you can get from the network, before it's been squashed down to apply to only the number of classes we are interested in. Check out how some researchers use them to train a shallow neural net based on what a deep network has learned: https://arxiv.org/pdf/1312.6184.pdf

It's kind of like how when learning a subject in detail, you will learn a great many minor points, but then when teaching a student, you will try to compress it to the simplest case. If the student now tried to teach, it'd be quite difficult, but would be able to describe it just well enough to use the language.

1

The logit (/ˈloʊdʒɪt/ LOH-jit) function is the inverse of the sigmoidal "logistic" function or logistic transform used in mathematics, especially in statistics. When the function's variable represents a probability p, the logit function gives the log-odds, or the logarithm of the odds p/(1 − p).

See here: https://en.wikipedia.org/wiki/Logit

1
  • 6
    That's in statistics/maths. We are talking machine learning here, where logit has different meaning. See this, this, this. Commented Jun 16, 2018 at 19:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.