0

The CRAN implementation of random forests offers both variable importance measures: the Gini importance as well as the widely used permutation importance defined as

For classification, it is the increase in percent of times a case is OOB and misclassified when the variable is permuted. For regression, it is the average increase in squared OOB residuals when the variable is permuted

By default h2o.varimp() computes only the former. Is there really no option in h2o to get the alternative measure out of a random forest model?

Thanks! ML

1 Answer 1

1

H2O does not calculate permutation importance. Please see the documentation for the explanation of how variable importance is calculated.

For your convenience I'll paste it as well below:

How is variable importance calculated for DRF?

Variable importance is determined by calculating the relative influence of each variable: whether that variable was selected during splitting in the tree building process and how much the squared error (over all trees) improved as a result.

A feature request has been previously made for this issue, you can follow it here (though note it is currently open).

3
  • Thanks, the answer is both useful and surprising since the Gini importance has been shown to suffer from enormous bias in the presence of catgeorical variables Aug 3, 2018 at 11:54
  • See this post (blog.hwr-berlin.de/codeandstats/…) for an example on the perils of Gini importance. Aug 11, 2018 at 6:39
  • May I ask if it is possible to obtain the oob indices for the individual trees in the h2o forests? That would enable me to write my own permutation importance function. Aug 24, 2018 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.