I am trying to figure out how to train a gbdt classifier with lightgbm in python, but getting confused with the example provided on the official website.
Following the steps listed, I find that the validation_data comes from nowhere and there is no clue about the format of the valid_data nor the merit or avail of training model with or without it.
Another question comes with it is that, in the documentation, it is said that "the validation data should be aligned with training data", while I look into the Dataset details, I find that there is another statement shows that "If this is Dataset for validation, training data should be used as reference".
My final questions are, why should validation data be aligned with training data? what is the meaning of reference in Dataset and how is it used during training? is the alignment goal accomplished with reference set to training data? what is the difference between this "reference" strategy and cross-validation?
Hope someone could help me out of this maze, thanks!
2 Answers
The idea of "validation data should be aligned with training data" is simple : every preprocessing you do to the training data, you should do it the same way for validation data and in production of course. This apply to every ML algorithm.
For example, for neural network, you will often normalize your training inputs (substract by mean and divide by std). Suppose you have a variable "age" with mean 26yo in training. It will be mapped to "0" for the training of your neural network. For validation data, you want to normalize in the same way as training data (using mean of training and std of training) in order that 26yo in validation is still mapped to 0 (same value -> same prediction).
This is the same for LightGBM. The data will be "bucketed" (in short, every continuous value will be discretized) and you want to map the continuous values to the same bins in training and in validation. Those bins will be calculated using the "reference" dataset.
Regarding training without validation, this is something you don't want to do most of the time! It is very easy to overfit the training data with boosted trees if you don't have a validation to adjust parameters such as "num_boost_round".
-
thank you @Florian Mutel, now I get the point of the reference parameter, but I think this has nothing to do with training itself, it is used only when we want to predict new samples, right? Do you mean that the lightGBM.train() method includes both training and predicting behavior when we provide valid_set?– kuixiongJun 28, 2019 at 10:08
-
again @FlorianMutel, this train method would only do one-pass validation while in fact we could conduct k-fold cross-validation using train method without providing valid_set and using predict method with valid_set on our own to validate the model? another big problem is that I find no 'reference' parameter with the predict method? how is this possible?– kuixiongJun 28, 2019 at 10:09
-
1The "reference" won't change the training of your model at all, it is useful for the creation of the validation dataset, which is then used when lightgbm calculates validation performance (after each boosting round). For Cross Validation, instead of coding it yourself, take a look at lightgbm.cv function. For prediction, there is no bucketing anymore: after training, the model is post-processed such as lightgbm replaces the bins by their boundaries values. It is much more convenient for prediction (no reference anymore) and for interpretation of the model (age > 26yo instead of age > bin122) Jun 28, 2019 at 11:48
still everything is tricky can you share full example with using and without using this "reference=" for example will it be different
import lightgbm as lgbm
importance_type_LGB = 'gain'
d_train = lgbm.Dataset(train_data_with_NANs, label= target_train)
d_valid = lgbm.Dataset(train_data_with_NANs, reference= target_train)
lgb_clf = lgbm.LGBMClassifier(class_weight = 'balanced' ,importance_type = importance_type_LGB)
lgb_clf.fit(test_data_with_NANs,target_train)
test_data_predict_proba_lgb = lgb_clf.predict_proba(test_data_with_NANs)
from
import lightgbm as lgbm
importance_type_LGB = 'gain'
lgb_clf = lgbm.LGBMClassifier(class_weight = 'balanced' ,importance_type = importance_type_LGB)
lgb_clf.fit(test_data_with_NANs,target_train)
test_data_predict_proba_lgb = lgb_clf.predict_proba(test_data_with_NANs)