16

I want to create a Pipeline in Scikit-Learn with a specific step being outlier detection and removal, allowing the transformed data to be passed to other transformers and estimator.

I have searched SE but can't find this answer anywhere. Is this possible?

6
  • Need more details. Is this a supervised or unsupervised task? The answer below provide some guidelines but only removes the outliers from X and not from y which you may need to consider depending on the task. Please elaborate more. Commented Sep 17, 2018 at 9:34
  • @VivekKumar your comment implies that there is no general procedure that works in both cases to this question. Whilst my below solution, as you point out, is specific to an unsupervised task (as was my problem), I would suppose that an extension to the code is possible and would account for both scenarios simultaneously. In which case I would challenge the downgrade to the question and instead suggest it references the answer instead.
    – Attack68
    Commented Sep 17, 2018 at 9:45
  • Yes that is definitely possible. But for that you will need to change the Pipeline too. Because currently, the scikit learn pipeline does not change y anywhere, and only pass that to next transformers. . Commented Sep 17, 2018 at 9:47
  • 3
    You can have a look at the source code of imblearn.Pipeline, which extends the sklearn pipeline to transform both X and y in it. Commented Sep 17, 2018 at 9:49
  • @VivekKumar very helpful, I have learnt something useful and I am sure anyone else stumbling on this question/answer will find it of use.
    – Attack68
    Commented Sep 17, 2018 at 9:50

1 Answer 1

12

Yes. Subclass the TransformerMixin and build a custom transformer. Here is an extension to one of the existing outlier detection methods:

from sklearn.pipeline import Pipeline, TransformerMixin
from sklearn.neighbors import LocalOutlierFactor

class OutlierExtractor(TransformerMixin):
    def __init__(self, **kwargs):
        """
        Create a transformer to remove outliers. A threshold is set for selection
        criteria, and further arguments are passed to the LocalOutlierFactor class

        Keyword Args:
            neg_conf_val (float): The threshold for excluding samples with a lower
               negative outlier factor.

        Returns:
            object: to be used as a transformer method as part of Pipeline()
        """

        self.threshold = kwargs.pop('neg_conf_val', -10.0)

        self.kwargs = kwargs

    def transform(self, X, y):
        """
        Uses LocalOutlierFactor class to subselect data based on some threshold

        Returns:
            ndarray: subsampled data

        Notes:
            X should be of shape (n_samples, n_features)
        """
        X = np.asarray(X)
        y = np.asarray(y)
        lcf = LocalOutlierFactor(**self.kwargs)
        lcf.fit(X)
        return (X[lcf.negative_outlier_factor_ > self.threshold, :],
                y[lcf.negative_outlier_factor_ > self.threshold])

    def fit(self, *args, **kwargs):
        return self

Then create a pipeline as:

pipe = Pipeline([('outliers', OutlierExtraction()), ...])
4
  • 13
    It may happer that the OP has a supervised task at hand and will pass y also along with X. In which case, the above solution will not work because X is changed and y is not and hence will have different lengths. Commented Sep 17, 2018 at 9:35
  • 3
    you can use imblearn pipeline which manages the y and X to have the same length
    – skibee
    Commented Sep 21, 2020 at 10:18
  • the updates by Daniel show how y can be altered at the same time as X
    – Attack68
    Commented Feb 4, 2021 at 14:12
  • Also interesting if you especially want that step in classifier you can use github.com/scikit-learn/scikit-learn/issues/…
    – RichieK
    Commented Apr 14, 2021 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.