Questions tagged [outliers]
An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.
1,088
questions
0
votes
0
answers
13
views
How to remove Outliers from a pyspark dataframe using Z-Score
Is there any library to calculate Z-score in pyspark?
I want to remove outliers from a dataframe using z-score. It there any easy way to do that?
0
votes
0
answers
26
views
How to detect outliers in time series data of multiple individuals?
I have time series data of a thousand people.
I want to make a model that finds outliers in the time series of the remaining 200 people by training about 800 outlier labeled data.
The data consists of ...
0
votes
0
answers
27
views
Remove outlier in diagnoistic plot R
I am running a linear model and checking the diagnostic plot of the model with the plot (model) function. I find that there is one outlier (37) and I want to remove it from the model.
What should the ...
2
votes
1
answer
30
views
How to manually insert outliers in the dummy dataset
I have created a dummy dataset with this below code :
from datetime import datetime
import numpy as np
import pandas as pd
from faker import Faker
fake = Faker()
def make_workers() -> list:
...
0
votes
2
answers
24
views
Pandas: replacing outlier values based with median values grouped by a timeframe
I have a pandas DataFrame of hourly financial valuations with some outlier values. The outliers have already been calculated and flagged in one of the dataframe's columns.
So the dataframe looks ...
0
votes
0
answers
15
views
Removing outlier does not produce the removed results
So I have this function that I made using the outlier formula.
def remove_outlier_IQR(df):
q1 = df.quantile(0.25)
q3 = df.quantile(0.75)
inter_q = q3-q1
df_final = ~((df<(q1 - 1.5*inter_q)) | (df&...
-1
votes
0
answers
13
views
How to set quantile values for minimum and maximum?
Here is an image:
And I have a doubt about how to show outliers and how to deal with them.
0
votes
1
answer
25
views
Use dataframe column to subset entire dataframe and apply function row-wise (anomaly detection)
Suppose I have a table like this
code
neighbours
t_min
abr007
abr018, abr030, abr032
16.2
abr018
abr007, abr030, laz246
20.9
abr030
abr007, abr032, cmp015
21.9
...
...
...
It contains ...
0
votes
0
answers
34
views
How to consider the removal of "Outliers percentage" ? What is acceptable?
I have a dataset with 5964 observations. It has a few columns with 20% (1201 observations), 12% (725 observations), and 5% (346 observartions) outliers. If I remove them, it will affect my dataset. ...
2
votes
3
answers
66
views
AttributeError: 'list' object has no attribute 'dropna' (outlier)
So I am trying to remove the outlier by making a function :
def remove_outlier_IQR(data):
Q1 = data.quantile(0.25)
Q3 = data.quantile(0.75)
Inter_Q = Q3-Q1
df_final = [~((data<(Q1 - 1.5*Inter_Q)) |...
1
vote
0
answers
52
views
Include outliers in ggplot boxplot
I conducted some interviews and I wanted to create box plots with ggplot based on these interviews. I managed to create the box plots but I do not manage to include the outliers in the box plot. I ...
0
votes
0
answers
27
views
Python removing outliers from plotly scatter plot
I coloured my outliers already and they are shown in the plot. I can`t figure out how to remove the "weird" outliers from the plot?
Flagging the weirdoes
weirdoes_condition = final['median [...
0
votes
1
answer
50
views
How to run user defined function over a window in spark dataframe?
I am trying to detect the outliers from my spark dataframe.
Below is the data sample.
pressure
Timestamp
358.64
2022-01-01 00:00:00
354.98
2022-01-01 00:10:00
350.34
2022-01-01 00:20:00
429.69
...
0
votes
1
answer
52
views
Tidyverse filter outliers - in one pipe
I want to filter outliers in the tidyverseframe work in one pipe.
Outlier for this example is just defined as Q1 - 1.5 * IQR and Q3 + 1.5 * IQR.
Q1 being the 25 percentile and Q3 the 75% percentile. ...
0
votes
1
answer
42
views
How do I include outlier points in the boxplot min and max range?
For WT_C57BL6, the boxplot is reading the bottom points as outliers. I want the boxplot min range to be extended to those "outlier" points and see the different quartile and mean placement. ...
1
vote
0
answers
29
views
"Most outlier" feature
I am using the Sklearn implementation of Isolation Forest (IF) to detect outliers on a set of data of 20-30 features.
It is working very well, but I would like insight into which feature has the ...
1
vote
1
answer
31
views
How to correctly check if there's at least one outlier in a OHLC price series on Python? Pandas related
Say you have the following df called df_trading_pair which contains price series registered in the 30m timeframe:
Start Date Open Price High Price Low Price Close Price End Date
...
-1
votes
1
answer
27
views
Detecting anomalies among several thousand users
I have this issue where I record a daily entry for all users in my system (several thousands, even 100.000+). These entries have 3 main features, "date", "file_count", "...
0
votes
1
answer
27
views
Method to quantize a range of values to keep precision when signficant outliers are present in the data
Could you tell me please if there is a suitable quantizing method in the following case (preferrably implemented in python)?
There is an input range where majority of values are within +-2 std from ...
0
votes
1
answer
20
views
Cut continuous data by outliers
For example I have DataFrame
df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6, 7, 8, 9], 'b': [2, 2, 4, 3, 1000, 2000, 1, 500, 3]})
I need to cut by outliers and get these intervals: 1-4, 5-6, 7, 8, 9.
...
1
vote
1
answer
54
views
How to detect outliners from a list
I have a list with values. From this list I would like to get the outliners.
list_of_values = [2, 3, 100, 5, 53, 5, 4, 7]
def detect_outlier(data):
threshold= 3
mean_1 = np.mean(data)
...
1
vote
1
answer
31
views
Create line from list of points while ignoring outliers
I have a list of points that almost create a straight line (but they are not perfectly align on that line). I want to create a line that best describes those points.
For example, for points:
points = [...
1
vote
1
answer
25
views
Create Loop for Box Plots in R
I have a dataset that contains both numeric and categorical values. I am trying to create box plots to visually identify outliers for each numeric column in my dataset. The below code works to do this,...
0
votes
0
answers
42
views
Error in FUN() only defined on a data frame with all numeric-alike variables
I am new to R. I am trying to get the mean, minimum, maximum and SD of pa1min after removing outliers. So far, I've removed outliers just fine.
activityUpper <- quantile(brfss2$pa1min, 0.997, na.rm=...
0
votes
1
answer
26
views
lapply for outliers poisition
I'm using this code to find the outliers in the data frame.
out<-lapply(df[,-1], function(x) boxplot.stats(x)$out)
The first columns are just the names (that's why I'm skipping it). I've found ...
0
votes
2
answers
37
views
Pandas finding and replacing outliers based on a group of two columns
I'm having a bit of trouble finding outliers in a df based on groups and dates.
For exampe I have a df like and I would like to find and replace the outlier values (10 for the group A on date 2022-06-...
0
votes
0
answers
20
views
Getting rid of outliers and bloxplot
I am trying to get rid of my outliers. My dataset is quite large aas it has 846 object and 19 variables. But every time I try to Boxplot my vehicle dataset, I get this error.
Error in x[floor(d)] + x[...
1
vote
1
answer
46
views
Outlier Detection on common variable name in R
I've would like to create a column to detect outliers in a large dataset. A sample of the dataset is
Item
Cost
Book A
5.0
Book A
3.5
Book A
12.0
Book B
6.0
Book B
8.0
Book C
3.0
Book B
6.0
...
0
votes
0
answers
24
views
Removing outliers with NAs in dataframe
I want to remove outliers using the function I found below, created by Statology. Unfortunately, my dataset contains NAs. Because of this I get the following error:
Error in quantile.default(x, probs =...
0
votes
1
answer
52
views
undefined columns selected and cannot xtfrm data frame error
I am trying to write a code that checks for outliers based on IQR and change those respective values to "NA". So I wrote this:
dt <- rnorm(200)
dg <- rnorm(200)
dh <- rnorm(200)
l ...
1
vote
1
answer
39
views
Removing outliers in various columns without creating NAs in the whole row
I am trying to remove the outliers from various variables at the same time in my dataset but with the function used it seems that when it finds one outlier it turns the whole row into NA.
That´s a ...
0
votes
0
answers
12
views
Precondition analysis
I would like to know whether I need to test every variable on its distribution and outliers or just the metrical variables? I mean does it make sense to test nominal variables on these aspects?
Thanks ...
0
votes
1
answer
76
views
Exclude Outliers in a Bland Altman Plot in R
I would like adapt my Bland Altman plot since it doesn't look good.
I would like to exclude outliers or only include the Confidence Interval.
I used this code:
library(BlandAltmanLeh)
library(ggplot2)
...
1
vote
0
answers
58
views
Outliers Detection
The outliers in this example was generated by me
The graph bellow represents the readings from one device, in the same day over multiple weeks, normal values in purple, red values are outliers, which ...
0
votes
1
answer
27
views
pandas df.apply Including condition with NaN values
I'm trying to replace outliers and NaN values in my pandas.DataFrame with the mode of the series, using the apply method and a lambda function and filtering by a property. I've tried in three ...
0
votes
0
answers
19
views
removing outliers from a dataset (RNA_SEQ)
I have an RNA-seq matrix.
I want to eliminate genes with extreme values in different conditions.
I know that there are different methods to eliminate outliers such as the IQR; the z-score, or boxplot$...
0
votes
0
answers
32
views
How to print outliers using a for-loop in R?
I want to create an algorithm using a for-loop that returns the lowest outlier for each numeric (or integer) variable. Where the lowest outlier is defined by a boxplot i.e. : boxplot.stats(dataframe[[...
5
votes
0
answers
99
views
Evaluate multiple Isolation Forest estimators during GridSearchCV with custom scorer function
I have a sample of values that don't have a y target value. Actually, the X features (predictors) are all used to fit the Isolation Forest estimator. The goal is to identify which of those X-features ...
0
votes
0
answers
47
views
How to winsorize at specific percentiles in R using winsorize() function?
I'm trying to winsorize a variable at 1% and 99% percentile using the code below,
convertible_2$open_new <- winsorize(convertible_2$open, minval=quantile(convertible_2$open, 0.01),
maxval=quantile(...
0
votes
1
answer
24
views
Similar function to scipy.stats.zscore but base on another "sample"
I have 2 datasets which describe the same process and I expect the same general range of values. So I would like to do is use scipy.stats.zscore on the one dataset but instead of using the sample ...
0
votes
0
answers
15
views
Statistics: Identify outlier values
Good day,
I am new to programming in R. I want to identify outlier actual values from my data. I know that to identify an outlier you need Q1-15.IQR and Q3+1.5IQR. I need help with finding those ...
0
votes
0
answers
56
views
How to remove outliers from nonlinear regression curve? with 3-sigma limits?
I tried to remove outliers from nonlinear regression curve, and get the updated formula and error criteria. Someone suggested me to use 3-sigma limits, a statistical calculation where the data are ...
1
vote
1
answer
40
views
Romove outliers from stat_summary in ggplot2
I have this part of code to produce boxplot with my data:
p <- ggplot(meltData, aes(x=variable, y=value)) +
geom_boxplot()+ geom_boxplot(outlier.colour="red", outlier.shape=1,outlier....
0
votes
0
answers
40
views
Finding threshold for extreme rainfall events by finding percentiles of skewed precipitation data
Suppose I have hourly rainfall data (aggregated to daily) for 5 stations in a region (duration 1980-2021). I'm trying to find the frequency of rainfall events exceeding the 95th, 97th, and 99th ...
0
votes
0
answers
32
views
How do I filter outliers in test data based on z-scores of train data?
I have a train and test dataset. On the train dataset I detected and deleted outlier values, when their standard deviation is 5 times greater from the mean. If a z-score returned is larger than that, ...
0
votes
1
answer
64
views
How can I identify inconsistencies and outliers in a dataset in R
I have a big dataset with alot of columns, being most of them not numeric values. I need to find inconsistencies in the data as well as outliers and the part of obtaining inconsistencies would be easy ...
1
vote
2
answers
43
views
Identifying position outliers of a moving object in a numpy array
I have a data set of positions (e.g. the x- or y-position of a movable object).
The object moves over time, let's say linearly. The distance between consecutive positions is within a certain range (e....
0
votes
1
answer
110
views
remove outliers from multiple columns in R
I used below codes to identify outliers on different columns:
outliers_x1 <- boxplot(mydata$x1, plot=FALSE)$out
outliers_x4 <- boxplot(mydata$x4, plot=FALSE)$out
outliers_x6 <- boxplot(mydata$...
2
votes
1
answer
41
views
Is it possible to filter out outliers that overlap based on time variable in R?
How do I delete outliers by checking for an overlap between ID column and date/time columns withfilter()?
For example, rows with ID =1 overlap in time as shown in the first 2 rows below,thus need to ...
0
votes
0
answers
27
views
ipython-input-9-69155119dc7e>:1: FutureWarning: Automatic reindexing on DataFrame
ipython-input-9-69155119dc7e>:1: FutureWarning: Automatic reindexing on DataFrame vs Series comparisons is deprecated and will raise ValueError in a future version. Do left, right = left.align(...