116

I've got some multivariate data of beauty vs ages. The ages range from 20-40 at intervals of 2 (20, 22, 24....40), and for each record of data, they are given an age and a beauty rating from 1-5. When I do boxplots of this data (ages across the X-axis, beauty ratings across the Y-axis), there are some outliers plotted outside the whiskers of each box.

I want to remove these outliers from the data frame itself, but I'm not sure how R calculates outliers for its box plots. Below is an example of what my data might look like. enter image description here

3
  • 3
    The boxplot function returns the outliers (among other statistics) invisibly. Try foo <- boxplot(...); foo and read ?boxplot to understand the output. Jan 24, 2011 at 21:37
  • 30
    Relevant: davidmlane.com/ben/outlier.gif
    – eyjo
    Jan 24, 2011 at 23:05
  • 1
    Can you send a link to the data? Mar 2, 2017 at 23:28

12 Answers 12

166

Nobody has posted the simplest answer:

x[!x %in% boxplot.stats(x)$out]

Also see this: http://www.r-statistics.com/2011/01/how-to-label-all-the-outliers-in-a-boxplot/

6
  • 5
    Really elegant. Thanks. But need to be careful if distribution has more than one mode and outliers are indeed only few and scattered.
    – KarthikS
    Mar 19, 2015 at 4:44
  • 1
    It would have been great if you were able to get index of them in a dataset. The way you are done will filter based on data value. If box plot is also doing grouping, not necessarily same data value will be outlier in each group
    – adam
    Jun 12, 2015 at 9:04
  • 3
    It's also important to mention that it does not change the dataset. This is just a filtering method. So if you intend to use the dataset without outliers assign it to a variable. e.g. result = x[!x %in% boxplot.stats(x)$out] May 7, 2017 at 18:17
  • @adam please see my answer to address this issue below. Dec 15, 2022 at 23:45
  • A question on boxplot.stats(): does anyone know how the function defines an outlier? E.g., based on a certain multiple of standard deviations from the mean?
    – bt3
    Jan 2, 2023 at 23:10
137

OK, you should apply something like this to your dataset. Do not replace & save or you'll destroy your data! And, btw, you should (almost) never remove outliers from your data:

remove_outliers <- function(x, na.rm = TRUE, ...) {
  qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)
  H <- 1.5 * IQR(x, na.rm = na.rm)
  y <- x
  y[x < (qnt[1] - H)] <- NA
  y[x > (qnt[2] + H)] <- NA
  y
}

To see it in action:

set.seed(1)
x <- rnorm(100)
x <- c(-10, x, 10)
y <- remove_outliers(x)
## png()
par(mfrow = c(1, 2))
boxplot(x)
boxplot(y)
## dev.off()

And once again, you should never do this on your own, outliers are just meant to be! =)

EDIT: I added na.rm = TRUE as default.

EDIT2: Removed quantile function, added subscripting, hence made the function faster! =)

enter image description here

5
  • 3
    OK, I'm missing something here. You want to remove outliers from data, so you can plot them with boxplot. That's manageable, and you should mark @Prasad's answer then, since answered your question. If you want to exclude outliers by using "outlier rule" q +/- (1.5 * H), hence run some analysis, then use this function. BTW, I did this from scratch, w/o Googling, so there's a chance that I've reenvented the wheel with this function of mine...
    – aL3xa
    Jan 25, 2011 at 0:27
  • To be even more general, make the 1.5 an arg of the function, with a default. And allow separate lower and upper ranges: c(-1.5,+1.5)
    – smci
    Apr 14, 2015 at 23:27
  • 8
    "outliers are just meant to be"? Not necessarily. They may come from measure errors, and must be thoroughly reviewed. When the outlier is too big, it may mean something, or not so much. That's why (at least in biology) the median usually says more about a population than the mean.
    – Rodrigo
    Feb 11, 2017 at 3:17
  • Nice. How about replace IQR with SD? e.g., H <- 8 * sd(x, na.rm = na.rm)? Would this exclude values above or below 8 sd of mean?
    – Aby
    May 31, 2017 at 8:14
  • indeed it would Jul 30, 2018 at 14:23
32

Use outline = FALSE as an option when you do the boxplot (read the help!).

> m <- c(rnorm(10),5,10)
> bp <- boxplot(m, outline = FALSE)

enter image description here

2
  • 4
    indeed, this will remove the outliers from the boxplot itself, but I want to remove the outliers from the data frame.
    – Dan Q
    Jan 24, 2011 at 21:53
  • 2
    I see, then as @Joshua said you need to look at the data returned by the boxplot function (in particular the out and group items in the list). Jan 24, 2011 at 21:55
17

The boxplot function returns the values used to do the plotting (which is actually then done by bxp():

bstats <- boxplot(count ~ spray, data = InsectSprays, col = "lightgray") 
#need to "waste" this plot
bstats$out <- NULL
bstats$group <- NULL
bxp(bstats)  # this will plot without any outlier points

I purposely did not answer the specific question because I consider it statistical malpractice to remove "outliers". I consider it acceptable practice to not plot them in a boxplot, but removing them just because they exceed some number of standard deviations or some number of inter-quartile widths is a systematic and unscientific mangling of the observational record.

4
  • 5
    Well, sidestepping the question without knowing why the question was asked is not a good practice either. Yes, it is not good to remove 'outliers' from the data but sometimes you need the data without outliers for specific tasks. In an statistics assignment I had recently, we had to visualise a set without its outliers to determine the best regression model to use for the data. So there! Jun 25, 2012 at 19:15
  • 4
    I'm not considering the advice you may have gotten in this regard to "determine the best regression model" to be particularly persuasive. Instead, if you needed to remove outliers for that vaguely stated purpose, then I think it reflects poorly on the persons who advised it rather than being evidence of invalidity of my position.
    – IRTFM
    Jun 25, 2012 at 19:25
  • I guess its legit when you know you're removing "noise". especially in physiological data.
    – roscoe1895
    Mar 29, 2019 at 8:04
  • Yes. If you have good reason to believe that a separate process creates the signal, it is justification for removal from the data.
    – IRTFM
    Mar 29, 2019 at 14:19
13

I looked up for packages related to removing outliers, and found this package (surprisingly called "outliers"!): https://cran.r-project.org/web/packages/outliers/outliers.pdf
if you go through it you see different ways of removing outliers and among them I found rm.outlier most convenient one to use and as it says in the link above: "If the outlier is detected and confirmed by statistical tests, this function can remove it or replace by sample mean or median" and also here is the usage part from the same source:
"Usage

rm.outlier(x, fill = FALSE, median = FALSE, opposite = FALSE)

Arguments
x a dataset, most frequently a vector. If argument is a dataframe, then outlier is removed from each column by sapply. The same behavior is applied by apply when the matrix is given.
fill If set to TRUE, the median or mean is placed instead of outlier. Otherwise, the outlier(s) is/are simply removed.
median If set to TRUE, median is used instead of mean in outlier replacement. opposite if set to TRUE, gives opposite value (if largest value has maximum difference from the mean, it gives smallest and vice versa) "

1
  • This seems great, but if you have a time series column in your data frame, it changes the time series.
    – PM0087
    Oct 15, 2019 at 8:47
9
x<-quantile(retentiondata$sum_dec_incr,c(0.01,0.99))
data_clean <- data[data$attribute >=x[1] & data$attribute<=x[2],]

I find this very easy to remove outliers. In the above example I am just extracting 2 percentile to 98 percentile of attribute values.

8

Wouldn't:

z <- df[df$x > quantile(df$x, .25) - 1.5*IQR(df$x) & 
        df$x < quantile(df$x, .75) + 1.5*IQR(df$x), ] #rows

accomplish this task quite easily?

0
6

Adding to @sefarkas' suggestion and using quantile as cut-offs, one could explore the following option:

newdata <- subset(mydata,!(mydata$var > quantile(mydata$var, probs=c(.01, .99))[2] | mydata$var < quantile(mydata$var, probs=c(.01, .99))[1]) ) 

This will remove the points points beyond the 99th quantile. Care should be taken like what aL3Xa was saying about keeping outliers. It should be removed only for getting an alternative conservative view of the data.

2
  • is it 0.91 or 0.99? as in mydata$var < quantile(mydata$var, probs=c(.01, .91))[1]) or mydata$var < quantile(mydata$var, probs=c(.01, .99))[1]) Apr 13, 2017 at 20:20
  • If you have a specific reason to use 91st percentile instead of 99th percentile, you can use it. It is only a heuristic
    – KarthikS
    Apr 16, 2017 at 0:44
3

1 way to do that is

my.NEW.data.frame <- my.data.frame[-boxplot.stats(my.data.frame$my.column)$out, ]

or

my.high.value <- which(my.data.frame$age > 200 | my.data.frame$age < 0) 
my.NEW.data.frame <- my.data.frame[-my.high.value, ]
1

Outliers are quite similar to peaks, so a peak detector can be useful for identifying outliers. The method described here has quite good performance using z-scores. The animation part way down the page illustrates the method signaling on outliers, or peaks.

Peaks are not always the same as outliers, but they're similar frequently.

An example is shown here: This dataset is read from a sensor via serial communications. Occasional serial communication errors, sensor error or both lead to repeated, clearly erroneous data points. There is no statistical value in these point. They are arguably not outliers, they are errors. The z-score peak detector was able to signal on spurious data points and generated a clean resulting dataset: enter image description here

0

Try this. Feed your variable in the function and save the o/p in the variable which would contain removed outliers

outliers<-function(variable){
    iqr<-IQR(variable)
    q1<-as.numeric(quantile(variable,0.25))
    q3<-as.numeric(quantile(variable,0.75))
    mild_low<-q1-(1.5*iqr)
    mild_high<-q3+(1.5*iqr)
    new_variable<-variable[variable>mild_low & variable<mild_high]
    return(new_variable)
}
1
  • Please add some explanation to your answer. Refer to How to answer.
    – ejderuby
    Aug 14, 2019 at 16:06
0

It is more difficult to remove outliers with grouped data because there is a risk of removing data points that are considered outliers in one group but not in others.

Because no dataset is provided I assume that there is a dependent variable "attractiveness", and two independent variables "age" and "gender". The boxplot shown in the original post above is then created with boxplot(dat$attractiveness ~ dat$gender + dat$age). To remove outliers you can use the following approach:

# Create a separate dataset for each group
group_data = split(dat, list(dat$age, dat$gender))

# Remove outliers from each dataset
group_data = lapply(group_data, function(x) {

  # Extract outlier values from boxplot
  outliers = boxplot.stats(x$attractiveness)$out

  # Remove outliers from data
  return(subset(x, !x$attractiveness %in% outliers))
})

# Combine datasets into a single dataset
dat = do.call(rbind, group_data)

Not the answer you're looking for? Browse other questions tagged or ask your own question.