3

So I am relatively new to the ML/AI game in python, and I'm currently working on a problem surrounding the implementation of a custom objective function for XGBoost.

My differential equation knowledge is pretty rusty so I've created a custom obj function with a gradient and hessian that models the mean squared error function that is ran as the default objective function in XGBRegressor to make sure that I am doing all of this correctly. The problem is, the results of the model (the error outputs are close but not identical for the most part (and way off for some points). I don't know what I'm doing wrong or how that could be possible if I am computing things correctly. If you all could look at this an maybe provide insight into where I am wrong, that would be awesome!

The original code without a custom function is:

    import xgboost as xgb

    reg = xgb.XGBRegressor(n_estimators=150, 
                   max_depth=2,
                   objective ="reg:squarederror", 
                   n_jobs=-1)

    reg.fit(X_train, y_train)

    y_pred_test = reg.predict(X_test)

and my custom objective function for MSE is as follows:

    def gradient_se(y_true, y_pred):
        #Compute the gradient squared error.
        return (-2 * y_true) + (2 * y_pred)

    def hessian_se(y_true, y_pred):
        #Compute the hessian for squared error
        return 0*(y_true + y_pred) + 2

   def custom_se(y_true, y_pred):
        #squared error objective. A simplified version of MSE used as
        #objective function.

        grad = gradient_se(y_true, y_pred)
        hess = hessian_se(y_true, y_pred)
        return grad, hess

the documentation reference is here

Thanks!

1 Answer 1

7

According to the documentation, the library passes the predicted values (y_pred in your case) and the ground truth values (y_true in your case) in this order.

You pass the y_true and y_pred values in reversed order in your custom_se(y_true, y_pred) function to both the gradient_se and hessian_se functions. For the hessian it doesn't make a difference since the hessian should return 2 for all x values and you've done that correctly.

For the gradient_se function you've incorrect signs for y_true and y_pred.

The correct implementation is as follows:

    def gradient_se(y_pred, y_true):
        #Compute the gradient squared error.
        return 2*(y_pred - y_true)

    def hessian_se(y_pred, y_true):
        #Compute the hessian for squared error
        return 0*y_true + 2

   def custom_se(y_pred, y_true):
        #squared error objective. A simplified version of MSE used as
        #objective function.

        grad = gradient_se(y_pred, y_true)
        hess = hessian_se(y_pred, y_true)
        return grad, hess

Update: Please keep in mind that the native XGBoost implementation and the implementation of the sklearn wrapper for XGBoost use a different ordering of the arguments. The native implementation takes predictions first and true labels (dtrain) second, while the sklearn implementation takes the true labels (dtrain) first and the predictions second.

9
  • According to the documentation, the order is y_true, ypred. See the Note. xgboost.readthedocs.io/en/latest/python/… But I think the docs are wrong, because all the objective functions implemented here flip the order github.com/dmlc/xgboost/blob/master/demo/guide-python/… Jul 12, 2021 at 16:32
  • Actually, from my experimentation, I'm seeing that y_true, y_pred is the correct ordering now. Jul 12, 2021 at 17:59
  • @PavelKomarov both the docs and the sample code show that preds is first and dtrain is second (dtrain has the labels), see github.com/dmlc/xgboost/blob/…
    – Qqbt
    Jul 23, 2021 at 8:32
  • 1
    I don't know what to tell you, bub. I printed the info coming in to the function, and it's clear the first argument holds the true targets, while the second is the model predictions (way off targets at first, but get closer). I'm using the sklearn wrapper, XGBRegressor, and I'm seeing both args come in as numpy arrays, rather than preds as an xgboost data matrix. Jul 24, 2021 at 10:53
  • Oh, you didn't mention that you're using the sklearn wrapper! The sklearn wrapper flips the arguments github.com/dmlc/xgboost/blob/…. Keep in mind that the original question is NOT using the sklearn wrapper and with the original code it was passing the arguments incorrectly, since it was not using the sklearn wrapper interface, so your particular comment here is just plain wrong, bub.
    – Qqbt
    Jul 25, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.