3

I'm using a logistic regression model in sklearn and I am interested in retrieving the log likelihood for such a model, so to perform an ordinary likelihood ratio test as suggested here.

The model is using the log loss as scoring rule. In the documentation, the log loss is defined "as the negative log-likelihood of the true labels given a probabilistic classifier’s predictions". However, the value is always positive, whereas the log likelihood should be negative. As an example:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

lr = LogisticRegression()
lr.fit(X_train, y_train)
y_prob = lr.predict_proba(X_test)
log_loss(y_test, y_prob)    # 0.66738

I do not see any method in the documentation for the model, is there any other possibility that I am currently not aware of?

2 Answers 2

9

Read closely; the log loss is the negative log-likelihood. Since log-likelihood is indeed as you say negative, its negative will be a positive number.

Let's see an example with dummy data:

from sklearn.metrics import log_loss
import numpy as np

y_true = np.array([0, 1, 1])
y_pred = np.array([0.1, 0.2, 0.9])

log_loss(y_true, y_pred)
# 0.60671964791658428

Now, let's compute manually the log-likelihood elements (i.e. one value per label-prediction pair), using the formula given in the scikit-learn docs you have linked to without the minus sign:

log_likelihood_elements = y_true*np.log(y_pred) + (1-y_true)*np.log(1-y_pred)
log_likelihood_elements
# array([-0.10536052, -1.60943791, -0.10536052])

Now, given the log-likelihood elements (which are indeed negative), the log loss is the negative of their sum, divided by the number of samples:

-np.sum(log_likelihood_elements)/len(y_true)
# 0.60671964791658428

log_loss(y_true, y_pred) == -np.sum(log_likelihood_elements)/len(y_true)
# True
3
  • Thank you so much! I am pretty stupid today. That means, the lower the better when using the log loss. Or I might change sign and treat it as log likelihood - right? Jan 10, 2018 at 11:30
  • 4
    @MattiaPaterna Every quantity described as "loss", implies "the lower the better"
    – desertnaut
    Jan 10, 2018 at 11:39
  • @Mattia Paterna: Cross-Entropy loss is negative log-likelihood because you are maximizing it in Maximum Likelyhood Estimation & use optimization algorithm minimization, thus in order to get maximization you need to multiply for -1
    – JeeyCi
    Mar 2 at 16:37
2

To get the log likelihood you can calculate:

-log_loss(y_true, y_pred)*len(y_true)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.