104

Can anyone explain the difference between the RandomForestClassifier and ExtraTreesClassifier in scikit learn. I've spent a good bit of time reading the paper:

P. Geurts, D. Ernst., and L. Wehenkel, “Extremely randomized trees”, Machine Learning, 63(1), 3-42, 2006

It seems these are the difference for ET:

1) When choosing variables at a split, samples are drawn from the entire training set instead of a bootstrap sample of the training set.

2) Splits are chosen completely at random from the range of values in the sample at each split.

The result from these two things are many more "leaves".

2
  • 7
    The reason I'm so interested in the extratreeclassifier is that I am getting much better results with ET on a particular problem. My feature vector is large >200 variables and the variables are very noisy. The standard RDF classifier gets lousy results but the ET is getting F1 scores of >90%. The classes are unbalanced with relatively few positive class samples and many negatives.
    – denson
    Mar 14, 2014 at 17:11
  • See also this more recent answer: stats.stackexchange.com/questions/175523/…
    – Archie
    Apr 23, 2018 at 15:01

3 Answers 3

64

Yes both conclusions are correct, although the Random Forest implementation in scikit-learn makes it possible to enable or disable the bootstrap resampling.

In practice, RFs are often more compact than ETs. ETs are generally cheaper to train from a computational point of view but can grow much bigger. ETs can sometime generalize better than RFs but it's hard to guess when it's the case without trying both first (and tuning n_estimators, max_features and min_samples_split by cross-validated grid search).

26

ExtraTrees classifier always tests random splits over fraction of features (in contrast to RandomForest, which tests all possible splits over fraction of features)

4
  • 14
    I'm amused that this comment is literally the word-for-word answer to a Coursera quiz question
    – Bob
    Jan 6, 2019 at 0:07
  • Yeah @Bob it is. I find this answer very useful that's why I posted here, it help other in understanding difference between extra-tree and random forest. Jan 7, 2019 at 8:52
  • 4
    also came from the same course. and this answer was helpful!
    – killezio
    Jan 21, 2019 at 14:50
  • yes @skeller88 this is amazing course. You should also look at this coursera.org/learn/competitive-data-science?specialization=aml Feb 28, 2020 at 20:01
0

The main difference between random forests and extra trees (usually called extreme random forests) lies in the fact that, instead of computing the locally optimal feature/split combination (for the random forest), for each feature under consideration, a random value is selected for the split (for the extra trees). Here is a good resource to know more about their difference in more detail Random forest vs extra tree.

1
  • 3
    I think you meant to paste a link. Oct 30, 2020 at 21:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.