12

I have trained a lightgbm model and I would like to plot the learning curves. How can I do that? In Keras for examples history returns the metrics so that I can plot them once training is over. How this task is handled here?

My code is the following:

def f_lgboost(data, params):

    model = lgb.LGBMClassifier(**params)


    X_train = data['X_train']

    y_train = data['y_train']

    X_dev = data['X_dev']

    y_dev = data['y_dev']

    X_test = data['X_test']

    categorical_feature= ['Ticker_code', 'Category_code']

    X_train[categorical_feature] = X_train[categorical_feature].astype('category')

    X_dev[categorical_feature] = X_dev[categorical_feature].astype('category')

    X_test[categorical_feature] = X_test[categorical_feature].astype('category')


    feature_name = X_train.columns.to_list()

    model.fit(X_train, y_train, eval_set = [(X_dev, y_dev)], eval_metric = 'auc', early_stopping_rounds = 20, 
              categorical_feature = categorical_feature, feature_name = feature_name)

    y_pred_train = model.predict_proba(X_train)[:, 1].ravel()

    y_pred_dev = model.predict_proba(X_dev)[:, 1].ravel()

    from sklearn.metrics import roc_auc_score

    auc_train = roc_auc_score(y_train, y_pred_train)

    auc_dev = roc_auc_score(y_dev, y_pred_dev)

    from sklearn.metrics import precision_recall_fscore_support

    precision, recall ,fscore, support = precision_recall_fscore_support(y_dev, (y_pred_dev > 0.5).astype(int), beta=0.5)

    y_pred_test = model.predict_proba(X_test)[:, 1].ravel()

    print(f'auc_train: {auc_train}, auc_dev : {auc_dev}, precision : {precision}, recall: {recall}, fscore : {fscore}')

    Results = {

            'params' : params,

            'data' : data,

            'lg_boost_model' : bst,

            'y_pred_train' : y_pred_train,

            'y_pred_dev' : y_pred_dev,

            'y_pred_test' : y_pred_test,

            'auc_train' : auc_train,

            'auc_dev' : auc_dev,

            'precision_dev': precision,

            'recall_dev' : recall,

            'fscore_dev' : fscore,

            'support_dev' : support


        }


    return Results

enter image description here

1 Answer 1

25

In the scikit-learn API, the learning curves are available via attribute lightgbm.LGBMModel.evals_result_. They will include metrics computed with datasets specified in the argument eval_set of method fit (so you would normally want to specify there both the training and the validation sets). There is also built-in plotting function, lightgbm.plot_metric, which accepts model.evals_result_ or model directly.

Here is a complete minimal example:

import lightgbm as lgb
import sklearn.datasets, sklearn.model_selection

X, y = sklearn.datasets.load_boston(return_X_y=True)
X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=7054)

model = lgb.LGBMRegressor(objective='mse', seed=8798, num_threads=1)
model.fit(X_train, y_train, eval_set=[(X_val, y_val), (X_train, y_train)], verbose=10)

lgb.plot_metric(model)

Here is the resulting plot:

Learning curves

4
  • How would you do this plot if you want to plot against epochs instead of iterations?
    – DataBach
    Commented Aug 16, 2021 at 14:47
  • They are, actually, epochs Commented Oct 18, 2021 at 18:39
  • I think this is similar to what I did, but plot__metric gives an error: TypeError: object of type 'NoneType' has no len()
    – Laurynas G
    Commented Nov 21, 2022 at 11:22
  • 1
    @LaurynasG check the object you passed to plot_metric(). Is it a fitted model or None? I assume you likely do model = model.fit(), and because model.fit() returns None, it causes the error.
    – Loc Quan
    Commented Aug 23 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.