25

I have a logistic regression and a random forest and I'd like to combine them (ensemble) for the final classification probability calculation by taking an average.

Is there a built-in way to do this in sci-kit learn? Some way where I can use the ensemble of the two as a classifier itself? Or would I need to roll my own classifier?

3
  • You need to roll your own, there's no way to combine two arbitrary classifiers.
    – Matti Lyra
    Commented Feb 3, 2014 at 16:32
  • 2
    There are several ongoing PRs and open issues on the sklearn github which are working towards having ensemble meta-estimators. Unfortunately none of them have been merged.
    – Daniel
    Commented Feb 4, 2014 at 3:47
  • @user1507844 could you take a stab at a similar question here ? stackoverflow.com/questions/23645837/…
    – ekta
    Commented May 14, 2014 at 5:00

4 Answers 4

34

NOTE: The scikit-learn Voting Classifier is probably the best way to do this now


OLD ANSWER:

For what it's worth I ended up doing this as follows:

class EnsembleClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self, classifiers=None):
        self.classifiers = classifiers

    def fit(self, X, y):
        for classifier in self.classifiers:
            classifier.fit(X, y)

    def predict_proba(self, X):
        self.predictions_ = list()
        for classifier in self.classifiers:
            self.predictions_.append(classifier.predict_proba(X))
        return np.mean(self.predictions_, axis=0)
8
  • 5
    Did you consider calibrating your estimators before averaging their prediction distributions? scikit-learn.org/stable/modules/calibration.html Commented Apr 22, 2015 at 13:13
  • Haven't tried that yet as it only came out in 0.16 but plan to try soon Commented Apr 22, 2015 at 17:02
  • I've tried calibrating, but at least for my specific problem, it actually made things worse... Commented May 8, 2015 at 5:32
  • 4
    @user1507844 You're probably getting worse performance because you're equally weighting all the classifiers' predictions. A better approach may be to try to minimize your loss function with a weight vector when combining the predictions. Look at the code here after line 50: kaggle.com/hsperr/otto-group-product-classification-challenge/… You could even optimize the hyperparameters of your individual classifiers using a package like hyperopt.github.io/hyperopt
    – Ryan
    Commented Jun 29, 2015 at 7:45
  • @Ryan that example code is not very useful. Mostly because algos has different ratio of train vs valid samples. For example, random forest can easily fit 100% of train data, and logistic regression could fit only 70%. On validation datasets they could give similar results, but the algo from the link above will greatly overwieght RF over LR Commented Jul 10, 2015 at 17:02
5

Given the same problem, I used a majority voting method. Combing probabilities/scores arbitrarily is very problematic, in that the performance of your different classifiers can be different, (For example, an SVM with 2 different kernels , + a Random forest + another classifier trained on a different training set).

One possible method to "weigh" the different classifiers, might be to use their Jaccard score as a "weight". (But be warned, as I understand it, the different scores are not "all made equal", I know that a Gradient Boosting classifier I have in my ensemble gives all its scores as 0.97, 0.98, 1.00 or 0.41/0 . I.E. it's very overconfident..)

5
  • 4
    Majority voting is fine for predicting which class an observation is in, but what if I want to know the probability of it being in that class? I'm fitting my individual classifiers to minimize log loss which I think avoids the "overconfidence" problem you describe. Commented Mar 2, 2014 at 16:18
  • 1
    The problem is with varying levels of performance by different predictors mainly. Commented Mar 2, 2014 at 16:33
  • 1
    I'm no expert but perhaps there is a way to weight the different predictors based on their performance. Is that what the Jaccard score you mention does? Commented Mar 2, 2014 at 20:03
  • The Jaccard score is a statistical score/performance metric. Like Accuracy, precision, recall, etc'. (Jaccard similarity coefficient score ) . Commented Mar 3, 2014 at 17:52
  • 2
    @user1507844: yes and (using stacking) those weights can be learned from a second-stage classifier (typically logistic regression, but could also be weighted averaging); moreover logistic regression gives more power than fixed weights; we can implicitly learn the specific cases where each classifier is good and bad. We train the level-2 classifier using both features + results from level-1 classifiers. Indeed you could even create level-2 (meta)features.
    – smci
    Commented Aug 18, 2015 at 20:59
5

What about the sklearn.ensemble.VotingClassifier?

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html#sklearn.ensemble.VotingClassifier

Per the description:

The idea behind the voting classifier implementation is to combine conceptually different machine learning classifiers and use a majority vote or the average predicted probabilities (soft vote) to predict the class labels. Such a classifier can be useful for a set of equally well performing model in order to balance out their individual weaknesses.

2
  • That didn't exist when I originally posted this question, but it is the proper sklearn implementation of my code I think. Great to see it in there now! Commented Dec 14, 2016 at 1:26
  • 1
    Excellent. I was wondering though after looking at it, if it would be possible to have differentent features for each classifier...
    – Gabriel
    Commented Dec 14, 2016 at 1:30
2

Now scikit-learn has StackingClassifier which can be used to stack multiple estimators.

from sklearn.datasets import load_iris  
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import StackingClassifier
X, y = load_iris(return_X_y=True)
estimators = [
    ('rf', RandomForestClassifier(n_estimators=10, random_state=42)),
    ('lg', LogisticRegression()))
   ]
clf = StackingClassifier(
estimators=estimators, final_estimator=LogisticRegression()
)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, random_state=42
)
clf.fit(X_train, y_train)
clf.predict_proba(X_test)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.