Questions tagged [missing-data]
For questions relating to missing data problems, which can involve special data structures, algorithms, statistical methods, modeling techniques, visualization, among other considerations.
2,723
questions
0
votes
0
answers
26
views
SQL query to find max value for each day available, and fill in rows for missing days to build a complete table?
I have a table from IBM Maximo called wostatus, which records changes in workorder statuses. There can be mutliple status changes in a single day, and changes are only recorded when they happen.
I ...
0
votes
0
answers
3
views
¿Why R wont knit my R Markdown archive? It says the data set does not exist in the current working directory, but its loaded and in files
enter image description here
Here I show the file loaded and in the "Files" place
Help please. Can sb tellme ¿Why R wont knit my R Markdown archive? It says the data set "...
2
votes
2
answers
74
views
Extend missing data in SELECT
I have following set od data:
ID
Date
Value
1
17.01.2023
Bad
1
17.01.2023
Good
1
18.01.2023
Enough
1
17.01.2023
Good
1
15.03.2023
Bad
1
20.03.2023
Good
2
10.01.2023
Bad
2
19.01.2023
Enough
2
...
1
vote
1
answer
17
views
Scikit learn Iterative Imputer : change and scaled tolerance
As part of a school project, I have to explore and perform data analysis and machine learning methods on a given database. The point is that my database is pretty big (12 651 lines for 810 columns) ...
0
votes
0
answers
6
views
Bitbucket webhook request payload for 'repo:refs_changed' miss 'changes'
I have a simple setup with bitbucket onprem and a webhook that makes an http request to a jenkins job on each "Push". On Jenkins, I have a Generic Job that picks up the json-payload and ...
0
votes
2
answers
34
views
I need to reassign values before missing to the last element (before the missing value) in SAS
For example let's say I have this data.
data have;
input end;
datalines;
1
2
4
5
.
4
8
6
.
1
2
3
.
;
I want it to look like this 5,5,5,5,.,6,6,6,.,3,3,3.
I've tried this.
data want;
set ...
0
votes
0
answers
13
views
How to get back categorical data after imputing using LabelEncoder + Iterative Imputer?
I am trying to impute missing values for a categorical column of data, i have successfully imputed them but now i want to change them back to categorical, how to do that?
i have used labelencoder and ...
0
votes
0
answers
8
views
Iterating a list with missing values: How do I replace the missing value with its previous existing value?
I am iterating two files with csv-values to join and compaire the matching values in a seperate file. In one file some of the values are missing and are saved as 'x'. I those missing values to be ...
0
votes
1
answer
48
views
I have no NAs in my dataset but I still get the error missing value where TRUE/FALSE needed?
I want to calculate the effect size of my variables. I am getting ther error "missing value wher TRUE/FALSE needed" even though I purged my data.frame of NAs before. Any idea why this is ...
0
votes
1
answer
43
views
Problem with pooling estimates from multiple imputed datasets using MICE in R- Zero-Inflated Poisson
I have been attempting to run a zero-inflated poisson regression on a dataframe that I have used mice() to impute missing data. My code successfully runs the multiple imputations and pools the results....
0
votes
0
answers
19
views
Does the IterativeImputer() work with categorical data?
I used fancyimpute and iterativeimputer() to impute missing values in my data set but the kernal keeps on running and it does not give an output. I don't know what is wrong.
I tried this code -
from ...
1
vote
1
answer
39
views
Replace missing values with the value of the column with the minimum sum of differences
I have the dataframe below.
# Create a sample DataFrame
df = pd.DataFrame({'Age': [np.nan, 31, 29, 43, np.nan],
'Weight': [np.nan, 100, 60, 75, np.nan],
'Height': ...
0
votes
1
answer
29
views
Returning only the unique values from two strings in Spotfire
I am trying to compare two columns a + b. Column A is populated with a string of data and Column B with another string of data. I am using the below formula in a custom column to return unique values ...
0
votes
2
answers
23
views
Table.Sort creating null fields in PowerQuery that once had data
I have the following code in PowerQuery
let
Source = #"MyDB",
_SomeTable = Source{[Schema="",Item="SomeTable"]}[Data],
#"Filtered Rows" = Table....
-3
votes
1
answer
35
views
How to make a date column in R show every hour in a year
I have a dataframe of information including hourly means for individual nodes, but when there is an hour missing it just skips that row.
How can I make it so that each node has the same number of rows,...
0
votes
0
answers
24
views
read.csv: some columns are set to zero (should be numeric but are imported as characters) in R [duplicate]
I am trying to import some csv files with field data to combine several csv files into one.
My goal is to have only numeric columns.
Unfortunately, R replaces all the values with "0" for ...
-1
votes
0
answers
23
views
Web Scraping missing value problem in python [duplicate]
I have created 4 empty list in a dict to store the the text of html tag named Product_name, Prices,
Description, Reviews when i run a for loop to scrap html text of different pages of flipkart i didn'...
0
votes
0
answers
34
views
Using stateful processing to identify and add missing timestamps in Apche Beam time series (Pcollection)
I want to add missing timestamps in a pcollection. My idea is:
Use a state to create a list of all available time steps
Infer from the list which time steps are missing
Create a pcollection with all ...
0
votes
1
answer
131
views
ImportError: cannot import name '_check_weights' from 'sklearn.neighbors._base'
I am trying to do Missforest as a method for handling missing values in table data.
import sklearn
print(sklearn.__version__)
->1.2.1
import sklearn.neighbors._base
import sys
sys.modules['sklearn....
0
votes
0
answers
25
views
Add missing time steps in a time series with Apache Beam
I am trying to add missing time steps (with a specific frequency) to a time series with Apache beam. The only solution I came up with was to use pandas and make a combineGlobally which is obviously ...
0
votes
2
answers
44
views
How to fill in a dataframe column based on another column of the same dataframe using a dictionary
I am working with a dataframe in Python with a 'Country Name' and a 'Region Name' column. The Country Name does not have any nans, while the Region Name has.
I have created a dictionary:
dict = {
&...
0
votes
2
answers
34
views
interpolate function does not fill nan values in data frame
Good evening, I have a problem filling in the missing data in a data frame.
I want to fill the ESG missing data of each company with the function interpolate if the number of missing values is between ...
1
vote
1
answer
23
views
How to fill in NA values according to a categorial variable in R
I have some NA values in my data that I would like to fill in according to the filename column. In order words, every observation with the same filename should have the same values in the discipline, ...
0
votes
2
answers
38
views
Different results from any(df.isnull()) and pd.isnull(data).any()
I am using the standard Boston houses data frame with pandas and I have noticed something that bugs me:
when I'm checking for missing values in 2 different ways - I'm getting 2 different results, ...
0
votes
2
answers
46
views
Third 3D coordinate of right angle triangle
For a right angle triangle in 3D space, if I have
Hypotenuse start and end 3D coordinates
3D plane definition
Lengths of two sides (other that hypotenuse) known
How can I calculate the 3rd 3D ...
0
votes
2
answers
30
views
Filling missing values along rows
I'm an absolute R beginner so please bear with me as I try to explain! I am looking for a code that will essentially fill in the NA values in the far right column (edu2019) by using the last available ...
2
votes
2
answers
43
views
Correlation testing with missing data
I want to test for sexual dependency in my data set, which consists of ordinal data. This means, that I have the sexes male (named as 1) and female (named as 2), and several traits (T1, T2, T3,...) of ...
1
vote
1
answer
20
views
Missing values on time variable, SPSS
I have problem with time variable (coded as hmm or hhmm) what is recorded as string. I need help with following:
Value 9999 has to be coded as missing, NA.
Value h99 or hh99, needs to be coded as h00 ...
0
votes
1
answer
69
views
How do you fill missing dates in a Polars dataframe (python)?
I do not seem to find an equivalent for Polars library. But basically, what I want to do is fill missing dates between two dates for a big dataframe. It has to be Polars because of the size of the ...
0
votes
0
answers
24
views
Losing year dummy when running 'within' plm in R
I'm working through question C3i in Wooldridge's Intro to Econometrics text and running the following code:
library(wooldridge)
library(plm)
jtrain.p <- pdata.frame(jtrain,index=c("fcode",...
0
votes
0
answers
12
views
Fill in Missing values of categorical nature that have been grouped by another column
I have two columns, one that has ratings for movies and another that has the genre of the movie. I have missing values in my rating column and i would like to fill in these missing values. The ratings ...
0
votes
0
answers
4
views
Ordination analysis of different sampling points based on physicochemical analyzes with NA values
For a research I'm conducting I need to analyze chemical data to "group" or "see how the sampling points lay out based on the chemical data".
This is the reproducible dataset:
Chem&...
-1
votes
1
answer
25
views
Replace value in one column with average of 7 columns
I have a dataframe containing hospital data. In one of my attribute i have a value of '-1' which i want to replace with the average value of 7 columns ahead it.
All tuples having Rating.Overall as '-...
0
votes
1
answer
20
views
impute missing values in a multi index data frame with taking the last available value of second index level
I'm struggling with the following problem:
I have a multilevel index data frame of time series data of following structure:
import pandas as pd
import numpy as np
multi_index = pd.MultiIndex....
1
vote
1
answer
36
views
How to include factor levels when they are missing in some replicas in R?
I have a dataframe df in which I have data about percentages (df$Percentage) of different habitats(df$Habitat) for specific tracks (df$Replica) in different sites (df$Site). As an example:
df <- ...
0
votes
1
answer
46
views
How to handle NaN values in rental price prediction project
I am working on a rental price prediction project where I web scraped data from Facebook Marketplace. When extracting the areas of the properties, I am encountering many NaN values.
I am web scraping ...
0
votes
3
answers
67
views
Remove columns with all NA values in multiple dataframes
I have >100 dataframes loaded into R with some columns containing 100% missing data which I would like to remove from all dataframes. In the below example with 3 dataframes, I would like to remove ...
1
vote
1
answer
52
views
NAs in a column that doesn't contain NAs R
I'm trying to understand what is happening to my code. Suppose I have data from this Google Forms and I want to calculate the percentages of a question with multiple options (namely, STRATEGY). There ...
-2
votes
1
answer
39
views
I want to create a list/array with missing values as element then, I wanted to fill that with some conditions
import pandas as pd
create a dataframe from the given data
#I wanted to create missing values for column-0
data_frame = pd.DataFrame({'column-0': ['Mumbai', '', 'Mumbai', 'Mumbai', 'Mumbai', 'Moscow', ...
0
votes
0
answers
40
views
Is there any way to fill in missing values other than making the data less fine?
I am trying to find a solution to the massive amount of missing data from the OceanAdapt Pinsky Northeast Spring American lobster data. Every pair of longitude and latitude has a max of 5 years of ...
0
votes
0
answers
29
views
replacing missing values in R with the one value that follows (not the mean)
I'm trying to replace the missing values in R with the value that follows, I have annual data for income by country, and for the missing income value for 2001 for country A I want it to pull the next ...
0
votes
0
answers
34
views
r circlize: missing value where TRUE/FALSE needed
I am trying to plot (for the first time) a chord diagram in the package circlize in R Studio. I am going through the manual chapters (Circular Visualization in R). The first step is to allocate the ...
0
votes
0
answers
32
views
How to generate missing values of three missing types(i.e. MCAR, MAR and MNAR) altogether using Python?
I have a complete dataset and now I want to generate missing values of three missing types(i.e. MCAR, MAR and MNAR) altogether.
If I generate each type of missing values one by one, will the ultimate ...
1
vote
2
answers
58
views
Fill missing values of dates using the number of the weekday
I have a dataset in long format. Every subject in the dataset was observed five times during the week. I have a column with the number of the day of the week in which the observation was supposed to ...
0
votes
0
answers
35
views
Correlation using listwise and pairwise deletion in R-correlations (Pearson)
I have a question regarding listwise & pairwise deletion in correlations.
If I use the functions complete.obs for listwise deletion and pairwise.complete.obs for pairwise deletion in a correlation ...
1
vote
1
answer
41
views
Add rows for missing data in a long dataset
I have a dataset in long format. Every subject in the dataset was observed a number of times (from 1 to 3) during the week. The observations occurred once a day from Monday to Friday based on the ...
0
votes
0
answers
39
views
"Calculating the coverage rate of the 100(1−α)% confidence intervals for average treatment effect"
I am using multiple imputation method to impute the missing covariates in my data set and then estimate the average treatment effects using these imputed covariates. I have R code (miATE) shared in ...
0
votes
2
answers
29
views
Classify a variable depending on the number of NAs in R
I have 2 scenarios:
One where I would like to define a new variable (called df$x1) depending on whether there are 16 NAs in 16 other different columns. My proposed code would be:
cols <- 1:16
df %...
0
votes
0
answers
32
views
Can Extra Trees regressor handling with missing values in Sklearn?
I am doing a project with sklearn.tree.ExtraTreeRegressor. It does not handle missing values. All tree-based algorithms handle the missing value internally. I have revised the info in the sklearn ...
0
votes
1
answer
45
views
What is the best practice to handle non-datetime timestamp column within pandas dataframe?
Let's say I have the following pandas dataframe with a non-standard timestamp column without datetime format. Due to I need to include a new column and convert it into an 24hourly-based timestamp for ...