5

The xgboost package allows to build a random forest (in fact, it chooses a random subset of columns to choose a variable for a split for the whole tree, not for a nod, as it is in a classical version of the algorithm, but it can be tolerated). But it seems that for regression only one tree from the forest (maybe, the last one built) is used.

To ensure that, consider just a standard toy example.

library(xgboost)
library(randomForest)
data(agaricus.train, package = 'xgboost')
    dtrain = xgb.DMatrix(agaricus.train$data,
 label = agaricus.train$label)
 bst = xgb.train(data = dtrain, 
                 nround = 1, 
                 subsample = 0.8, 
                 colsample_bytree = 0.5, 
                 num_parallel_tree = 100, 
                 verbose = 2, 
                 max_depth = 12)

answer1 = predict(bst, dtrain); 
(answer1 - agaricus.train$label) %*% (answer1 -  agaricus.train$label)

forest = randomForest(x = as.matrix(agaricus.train$data), y = agaricus.train$label, ntree = 50)

answer2 = predict(forest, as.matrix(agaricus.train$data))
(answer2 - agaricus.train$label) %*% (answer2 -  agaricus.train$label)

Yes, of course, the default version of the xgboost random forest uses not a Gini score function but just the MSE; it can be changed easily. Also it is not correct to do such a validation and so on, so on. It does not affect a main problem. Regardless of which sets of parameters are being tried results are suprisingly bad compared with the randomForest implementation. This holds for another data sets as well.

Could anybody provide a hint on such strange behaviour? When it comes to the classification task the algorithm does work as expected.

#

Well, all trees are grown and all are used to make a prediction. You may check that using the parameter 'ntreelimit' for the 'predict' function.

The main problem remains: is the specific form of the Random Forest algorithm that is produced by the xgbbost package valid?

Cross-validation, parameter tunning and other crap have nothing to do with that -- every one may add necessary corrections to the code and see what happens.

You may specify the 'objective' option like this:

mse = function(predict, dtrain)
{
  real = getinfo(dtrain, 'label')
  return(list(grad = 2 * (predict - real),
              hess = rep(2, length(real))))
}

This provides that you use the MSE when choosing a variable for the split. Even after that, results are suprisingly bad compared to those of randomForest.

Maybe, the problem is of academical nature and concerns the way how a random subset of features to make a split is chosen. The classical implementation chooses a subset of features (the size is specified with 'mtry' for the randomForest package) for EVERY split separately and the xgboost implementation chooses one subset for a tree (specified with 'colsample_bytree').

So this fine difference appears to be of great importance, at least for some types of datasets. It is interesting, indeed.

2
  • 1
    What is your question ?
    – eliasah
    Jan 19, 2016 at 12:40
  • As I have already said, I am wondering why one gets such results. Is there a mistake in the package? Or am I just having a mistake in how the package's functions are used?
    – mv_
    Jan 19, 2016 at 13:35

1 Answer 1

6

xgboost(random forest style) does use more than one tree to predict. But there are many other differences to explore.

I myself am new to xgboost, but curious. So I wrote the code below to visualize the trees. You can run the code yourself to verify or explore other differences.

Your data set of choice is a classification problem as labels are either 0 or 1. I like to switch to a simple regression problem to visualize what xgboost does.

true model: $y = x_1 * x_2$ + noise

If you train a single tree or multiple tree, with the code examples below you observe that the learned model structure does contain more trees. You cannot argue alone from the prediction accuracy how many trees are trained.

Maybe the predictions are different because the implementations are different. None of the ~5 RF implementations I know of are exactly alike, and this xgboost(rf style) is as closest a distant "cousin".

I observe the colsample_bytree is not equal to mtry, as the former uses the same subset of variable/columns for the entire tree. My regression problem is one big interaction only, which cannot be learned if trees only uses either x1 or x2. Thus in this case colsample_bytree must be set to 1 to use both variables in all trees. Regular RF could model this problem with mtry=1, as each node would use either X1 or X2

I see your randomForest predictions are not out-of-bag cross-validated. If drawing any conclusions on predictions you must cross-validate, especially for fully grown trees.

NB You need to fix the function vec.plot as does not support xgboost out of the box, because xgboost out of some other box do not take data.frame as an valid input. The instruction in the code should be clear

library(xgboost)
library(rgl)
library(forestFloor)
Data = data.frame(replicate(2,rnorm(5000)))
Data$y = Data$X1*Data$X2 + rnorm(5000)*.5
gradientByTarget =fcol(Data,3)
plot3d(Data,col=gradientByTarget) #true data structure

fix(vec.plot) #change these two line in the function, as xgboost do not support data.frame
#16# yhat.vec = predict(model, as.matrix(Xtest.vec))
#21# yhat.obs = predict(model, as.matrix(Xtest.obs))

#1 single deep tree
xgb.model =  xgboost(data = as.matrix(Data[,1:2]),label=Data$y,
                     nrounds=1,params = list(max.depth=250))
vec.plot(xgb.model,as.matrix(Data[,1:2]),1:2,col=gradientByTarget,grid=200)
plot(Data$y,predict(xgb.model,as.matrix(Data[,1:2])),col=gradientByTarget)
#clearly just one tree

#100 trees (gbm boosting)
xgb.model =  xgboost(data = as.matrix(Data[,1:2]),label=Data$y,
                     nrounds=100,params = list(max.depth=16,eta=.5,subsample=.6))
vec.plot(xgb.model,as.matrix(Data[,1:2]),1:2,col=gradientByTarget) 
plot(Data$y,predict(xgb.model,as.matrix(Data[,1:2])),col=gradientByTarget) ##predictions are not OOB cross-validated!


#20 shallow trees (bagging)
xgb.model =  xgboost(data = as.matrix(Data[,1:2]),label=Data$y,
                     nrounds=1,params = list(max.depth=250,
                     num_parallel_tree=20,colsample_bytree = .5, subsample = .5))
vec.plot(xgb.model,as.matrix(Data[,1:2]),1:2,col=gradientByTarget) #bagged mix of trees
plot(Data$y,predict(xgb.model,as.matrix(Data[,1:2]))) #terrible fit!!
#problem, colsample_bytree is NOT mtry as columns are only sampled once
# (this could be raised as an issue on their github page, that this does not mimic RF)


#20 deep tree (bagging), no column limitation
xgb.model =  xgboost(data = as.matrix(Data[,1:2]),label=Data$y,
                     nrounds=1,params = list(max.depth=500,
                     num_parallel_tree=200,colsample_bytree = 1, subsample = .5))
vec.plot(xgb.model,as.matrix(Data[,1:2]),1:2,col=gradientByTarget) #boosted mix of trees
plot(Data$y,predict(xgb.model,as.matrix(Data[,1:2])))
#voila model can fit data
3
  • Great thing about vizualization! 1. I was a bit too lazy to put here a proper code, with all validations and other stuff needded. It is not important for the question under consideration. In fact, even the cross-validation procedure must be repeated a few times (well, much more than a couple) to be totally sure that method works, which is often forgotten.2. About other stuff -- I made changes to the main post.
    – mv_
    Jan 21, 2016 at 12:13
  • I cannot get the vec.plot work even with the fix: Error in UseMethod("predict") : no applicable method for 'predict' applied to an object of class "xgb.Booster". edit: works when I change predict to xgboost::predict in vec.plot
    – spore234
    Feb 20, 2016 at 14:29
  • I ran the code and did not found the error. I guess you got some namespace error because your search path differed, try run search() . It should not be needed for a predict() method to specify package. Maybe the error would dissipate by restarting R and loading packages in the suggested order. But hey if it works, then fine :) Feb 21, 2016 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.