Questions tagged [missing-data]

For questions relating to missing data problems, which can involve special data structures, algorithms, statistical methods, modeling techniques, visualization, among other considerations.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

SQL query to find max value for each day available, and fill in rows for missing days to build a complete table?

I have a table from IBM Maximo called wostatus, which records changes in workorder statuses. There can be mutliple status changes in a single day, and changes are only recorded when they happen. I ...
  • 1
0 votes
0 answers
3 views

¿Why R wont knit my R Markdown archive? It says the data set does not exist in the current working directory, but its loaded and in files

enter image description here Here I show the file loaded and in the "Files" place Help please. Can sb tellme ¿Why R wont knit my R Markdown archive? It says the data set "...
2 votes
2 answers
74 views

Extend missing data in SELECT

I have following set od data: ID Date Value 1 17.01.2023 Bad 1 17.01.2023 Good 1 18.01.2023 Enough 1 17.01.2023 Good 1 15.03.2023 Bad 1 20.03.2023 Good 2 10.01.2023 Bad 2 19.01.2023 Enough 2 ...
  • 21
1 vote
1 answer
17 views

Scikit learn Iterative Imputer : change and scaled tolerance

As part of a school project, I have to explore and perform data analysis and machine learning methods on a given database. The point is that my database is pretty big (12 651 lines for 810 columns) ...
0 votes
0 answers
6 views

Bitbucket webhook request payload for 'repo:refs_changed' miss 'changes'

I have a simple setup with bitbucket onprem and a webhook that makes an http request to a jenkins job on each "Push". On Jenkins, I have a Generic Job that picks up the json-payload and ...
  • 1,144
0 votes
2 answers
34 views

I need to reassign values before missing to the last element (before the missing value) in SAS

For example let's say I have this data. data have; input end; datalines; 1 2 4 5 . 4 8 6 . 1 2 3 . ; I want it to look like this 5,5,5,5,.,6,6,6,.,3,3,3. I've tried this. data want; set ...
  • 13
0 votes
0 answers
13 views

How to get back categorical data after imputing using LabelEncoder + Iterative Imputer?

I am trying to impute missing values for a categorical column of data, i have successfully imputed them but now i want to change them back to categorical, how to do that? i have used labelencoder and ...
0 votes
0 answers
8 views

Iterating a list with missing values: How do I replace the missing value with its previous existing value?

I am iterating two files with csv-values to join and compaire the matching values in a seperate file. In one file some of the values are missing and are saved as 'x'. I those missing values to be ...
0 votes
1 answer
48 views

I have no NAs in my dataset but I still get the error missing value where TRUE/FALSE needed?

I want to calculate the effect size of my variables. I am getting ther error "missing value wher TRUE/FALSE needed" even though I purged my data.frame of NAs before. Any idea why this is ...
0 votes
1 answer
43 views

Problem with pooling estimates from multiple imputed datasets using MICE in R- Zero-Inflated Poisson

I have been attempting to run a zero-inflated poisson regression on a dataframe that I have used mice() to impute missing data. My code successfully runs the multiple imputations and pools the results....
  • 1
0 votes
0 answers
19 views

Does the IterativeImputer() work with categorical data?

I used fancyimpute and iterativeimputer() to impute missing values in my data set but the kernal keeps on running and it does not give an output. I don't know what is wrong. I tried this code - from ...
1 vote
1 answer
39 views

Replace missing values with the value of the column with the minimum sum of differences

I have the dataframe below. # Create a sample DataFrame df = pd.DataFrame({'Age': [np.nan, 31, 29, 43, np.nan], 'Weight': [np.nan, 100, 60, 75, np.nan], 'Height': ...
0 votes
1 answer
29 views

Returning only the unique values from two strings in Spotfire

I am trying to compare two columns a + b. Column A is populated with a string of data and Column B with another string of data. I am using the below formula in a custom column to return unique values ...
0 votes
2 answers
23 views

Table.Sort creating null fields in PowerQuery that once had data

I have the following code in PowerQuery let Source = #"MyDB", _SomeTable = Source{[Schema="",Item="SomeTable"]}[Data], #"Filtered Rows" = Table....
  • 551
-3 votes
1 answer
35 views

How to make a date column in R show every hour in a year

I have a dataframe of information including hourly means for individual nodes, but when there is an hour missing it just skips that row. How can I make it so that each node has the same number of rows,...
0 votes
0 answers
24 views

read.csv: some columns are set to zero (should be numeric but are imported as characters) in R [duplicate]

I am trying to import some csv files with field data to combine several csv files into one. My goal is to have only numeric columns. Unfortunately, R replaces all the values with "0" for ...
-1 votes
0 answers
23 views

Web Scraping missing value problem in python [duplicate]

I have created 4 empty list in a dict to store the the text of html tag named Product_name, Prices, Description, Reviews when i run a for loop to scrap html text of different pages of flipkart i didn'...
0 votes
0 answers
34 views

Using stateful processing to identify and add missing timestamps in Apche Beam time series (Pcollection)

I want to add missing timestamps in a pcollection. My idea is: Use a state to create a list of all available time steps Infer from the list which time steps are missing Create a pcollection with all ...
  • 231
0 votes
1 answer
131 views

ImportError: cannot import name '_check_weights' from 'sklearn.neighbors._base'

I am trying to do Missforest as a method for handling missing values in table data. import sklearn print(sklearn.__version__) ->1.2.1 import sklearn.neighbors._base import sys sys.modules['sklearn....
  • 91
0 votes
0 answers
25 views

Add missing time steps in a time series with Apache Beam

I am trying to add missing time steps (with a specific frequency) to a time series with Apache beam. The only solution I came up with was to use pandas and make a combineGlobally which is obviously ...
  • 231
0 votes
2 answers
44 views

How to fill in a dataframe column based on another column of the same dataframe using a dictionary

I am working with a dataframe in Python with a 'Country Name' and a 'Region Name' column. The Country Name does not have any nans, while the Region Name has. I have created a dictionary: dict = { &...
0 votes
2 answers
34 views

interpolate function does not fill nan values in data frame

Good evening, I have a problem filling in the missing data in a data frame. I want to fill the ESG missing data of each company with the function interpolate if the number of missing values is between ...
1 vote
1 answer
23 views

How to fill in NA values according to a categorial variable in R

I have some NA values in my data that I would like to fill in according to the filename column. In order words, every observation with the same filename should have the same values in the discipline, ...
  • 25
0 votes
2 answers
38 views

Different results from any(df.isnull()) and pd.isnull(data).any()

I am using the standard Boston houses data frame with pandas and I have noticed something that bugs me: when I'm checking for missing values in 2 different ways - I'm getting 2 different results, ...
  • 51
0 votes
2 answers
46 views

Third 3D coordinate of right angle triangle

For a right angle triangle in 3D space, if I have Hypotenuse start and end 3D coordinates 3D plane definition Lengths of two sides (other that hypotenuse) known How can I calculate the 3rd 3D ...
  • 7
0 votes
2 answers
30 views

Filling missing values along rows

I'm an absolute R beginner so please bear with me as I try to explain! I am looking for a code that will essentially fill in the NA values in the far right column (edu2019) by using the last available ...
  • 5
2 votes
2 answers
43 views

Correlation testing with missing data

I want to test for sexual dependency in my data set, which consists of ordinal data. This means, that I have the sexes male (named as 1) and female (named as 2), and several traits (T1, T2, T3,...) of ...
  • 35
1 vote
1 answer
20 views

Missing values on time variable, SPSS

I have problem with time variable (coded as hmm or hhmm) what is recorded as string. I need help with following: Value 9999 has to be coded as missing, NA. Value h99 or hh99, needs to be coded as h00 ...
0 votes
1 answer
69 views

How do you fill missing dates in a Polars dataframe (python)?

I do not seem to find an equivalent for Polars library. But basically, what I want to do is fill missing dates between two dates for a big dataframe. It has to be Polars because of the size of the ...
0 votes
0 answers
24 views

Losing year dummy when running 'within' plm in R

I'm working through question C3i in Wooldridge's Intro to Econometrics text and running the following code: library(wooldridge) library(plm) jtrain.p <- pdata.frame(jtrain,index=c("fcode",...
  • 23
0 votes
0 answers
12 views

Fill in Missing values of categorical nature that have been grouped by another column

I have two columns, one that has ratings for movies and another that has the genre of the movie. I have missing values in my rating column and i would like to fill in these missing values. The ratings ...
  • 1
0 votes
0 answers
4 views

Ordination analysis of different sampling points based on physicochemical analyzes with NA values

For a research I'm conducting I need to analyze chemical data to "group" or "see how the sampling points lay out based on the chemical data". This is the reproducible dataset: Chem&...
-1 votes
1 answer
25 views

Replace value in one column with average of 7 columns

I have a dataframe containing hospital data. In one of my attribute i have a value of '-1' which i want to replace with the average value of 7 columns ahead it. All tuples having Rating.Overall as '-...
0 votes
1 answer
20 views

impute missing values in a multi index data frame with taking the last available value of second index level

I'm struggling with the following problem: I have a multilevel index data frame of time series data of following structure: import pandas as pd import numpy as np multi_index = pd.MultiIndex....
1 vote
1 answer
36 views

How to include factor levels when they are missing in some replicas in R?

I have a dataframe df in which I have data about percentages (df$Percentage) of different habitats(df$Habitat) for specific tracks (df$Replica) in different sites (df$Site). As an example: df <- ...
  • 1,254
0 votes
1 answer
46 views

How to handle NaN values in rental price prediction project

I am working on a rental price prediction project where I web scraped data from Facebook Marketplace. When extracting the areas of the properties, I am encountering many NaN values. I am web scraping ...
0 votes
3 answers
67 views

Remove columns with all NA values in multiple dataframes

I have >100 dataframes loaded into R with some columns containing 100% missing data which I would like to remove from all dataframes. In the below example with 3 dataframes, I would like to remove ...
1 vote
1 answer
52 views

NAs in a column that doesn't contain NAs R

I'm trying to understand what is happening to my code. Suppose I have data from this Google Forms and I want to calculate the percentages of a question with multiple options (namely, STRATEGY). There ...
-2 votes
1 answer
39 views

I want to create a list/array with missing values as element then, I wanted to fill that with some conditions

import pandas as pd create a dataframe from the given data #I wanted to create missing values for column-0 data_frame = pd.DataFrame({'column-0': ['Mumbai', '', 'Mumbai', 'Mumbai', 'Mumbai', 'Moscow', ...
0 votes
0 answers
40 views

Is there any way to fill in missing values other than making the data less fine?

I am trying to find a solution to the massive amount of missing data from the OceanAdapt Pinsky Northeast Spring American lobster data. Every pair of longitude and latitude has a max of 5 years of ...
  • 1
0 votes
0 answers
29 views

replacing missing values in R with the one value that follows (not the mean)

I'm trying to replace the missing values in R with the value that follows, I have annual data for income by country, and for the missing income value for 2001 for country A I want it to pull the next ...
  • 1
0 votes
0 answers
34 views

r circlize: missing value where TRUE/FALSE needed

I am trying to plot (for the first time) a chord diagram in the package circlize in R Studio. I am going through the manual chapters (Circular Visualization in R). The first step is to allocate the ...
  • 576
0 votes
0 answers
32 views

How to generate missing values of three missing types(i.e. MCAR, MAR and MNAR) altogether using Python?

I have a complete dataset and now I want to generate missing values of three missing types(i.e. MCAR, MAR and MNAR) altogether. If I generate each type of missing values one by one, will the ultimate ...
1 vote
2 answers
58 views

Fill missing values of dates using the number of the weekday

I have a dataset in long format. Every subject in the dataset was observed five times during the week. I have a column with the number of the day of the week in which the observation was supposed to ...
0 votes
0 answers
35 views

Correlation using listwise and pairwise deletion in R-correlations (Pearson)

I have a question regarding listwise & pairwise deletion in correlations. If I use the functions complete.obs for listwise deletion and pairwise.complete.obs for pairwise deletion in a correlation ...
user avatar
1 vote
1 answer
41 views

Add rows for missing data in a long dataset

I have a dataset in long format. Every subject in the dataset was observed a number of times (from 1 to 3) during the week. The observations occurred once a day from Monday to Friday based on the ...
0 votes
0 answers
39 views

"Calculating the coverage rate of the 100(1−α)% confidence intervals for average treatment effect"

I am using multiple imputation method to impute the missing covariates in my data set and then estimate the average treatment effects using these imputed covariates. I have R code (miATE) shared in ...
  • 1
0 votes
2 answers
29 views

Classify a variable depending on the number of NAs in R

I have 2 scenarios: One where I would like to define a new variable (called df$x1) depending on whether there are 16 NAs in 16 other different columns. My proposed code would be: cols <- 1:16 df %...
  • 537
0 votes
0 answers
32 views

Can Extra Trees regressor handling with missing values in Sklearn?

I am doing a project with sklearn.tree.ExtraTreeRegressor. It does not handle missing values. All tree-based algorithms handle the missing value internally. I have revised the info in the sklearn ...
  • 1
0 votes
1 answer
45 views

What is the best practice to handle non-datetime timestamp column within pandas dataframe?

Let's say I have the following pandas dataframe with a non-standard timestamp column without datetime format. Due to I need to include a new column and convert it into an 24hourly-based timestamp for ...
  • 1,396

1
2 3 4 5
55