Questions tagged [data-cleaning]

Data cleaning is the process of removing or repairing errors, and normalizing data used in computer programs. For example, outliers may be removed, missing samples may be interpolated, invalid values may be marked as unavailable, and synonymous values may be merged. One approach to data cleaning is the "tidy data" framework from Wickham, which means each row is an observation and each column is a variable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

How to extract tabular data present in PDF file using python?

I want to extract tabular data from PDF file image below Statistical Import/Export Data of year 1992 to 1993 in PDF file The data I want is the numerical data to create a data set. Any Suggestions ...
khankhattak's user avatar
-1 votes
1 answer
30 views

Error filtering multiple csv files based on rows

I have a folder that contains 20 csv files. Each file has about 10 columns and thousands of rows. The csv files look something like the following: gene p-value xyz acan 0.05 123 mmp2 0.02 456 mmp9 ...
Adriana's user avatar
  • 93
0 votes
1 answer
32 views

R: length of vector not equal to column numbers

I have vectors with varying length, like 3,7,9,19etc. I’d like to use these vectors to fill in multiple columns in each row. I have 19 columns in total. When the vector length is smaller than 19, I’d ...
linda.h's user avatar
1 vote
2 answers
67 views

What gets passed to the mutate and modify?

I'm fairly new to R but not new to programming itself. I am using a simplified example of my code here. I have a dataframe that has three columns ( doc_id, tag_list, single_tag) all of which are ...
Akhil's user avatar
  • 182
0 votes
1 answer
33 views

Error when trying to create a list of files

I have a folder that contains 20 csv files. Each file has about 10 columns and thousands of rows. The csv files look something like the following: gene p-value xyz acan 0.05 123 mmp2 0.02 456 mmp9 ...
Adriana's user avatar
  • 93
-4 votes
0 answers
35 views

find all random letter entries in a database [closed]

I have a database with 3 (should be) numerical columns that has 645000 rows. However, there are entries with letters randomly, and I need to find the letters. I need to find all random entries that ...
Letty's user avatar
  • 1
-4 votes
0 answers
18 views

While data cleaning, how to decide which column to drop? [closed]

While data cleaning (as a data analytics intern), I have confusion about which column to drop, which column to fillna with 'unknown' and which column to not drop.. can you please give me suggestions ...
Prashant Sharma's user avatar
2 votes
3 answers
59 views

Python - Remove items with same name but different extension from a list

I have a list of pictures when for every picture in .jpg format I have one corresponding .mov file with the same name. I want to remove all .mov files that have a corresponding .jpg file only, for ...
Javi's user avatar
  • 61
0 votes
1 answer
40 views

Python - Remove files with same name but different extension

I am making an inventory of the pictures I have within a specific folder and I want to remove some unwanted pictures. The pictures I need to get rid of are .mov files that represent the live status of ...
Javi's user avatar
  • 61
0 votes
0 answers
6 views

Python TypeError: < not supported between instances of str and int in training model

I'm encountering an issue when trying to convert an object to a string in Python. Specifically, I'm working with a dataset for a Named Entity Recognition (NER) task (new at this), and I need to ...
Manuela Str's user avatar
0 votes
0 answers
8 views

How to bulk delete duplicates in erpnext

How to bulk delete duplicates in erpnext I tried doing this in data import>update existing records but the downloaded file is 20mb and allowed import file size is 10mb I want to know how we can ...
ram kumar's user avatar
0 votes
2 answers
27 views

How to remove extra values from a column in R

I am working with a dataset, and in the date column some of the values have letters as well. It's a large dataset, so I would need to be able to apply this to the entire column. Example: 'XVII) (2016' ...
Teresa Dieter's user avatar
0 votes
1 answer
25 views

Is there an easy way to get the transpose of a data frame while grouping the values of one of their columns? [duplicate]

I have the following data frame: df <- data.frame(group = c("A","A","A","B","B","C","C","C"), year = c(&...
Ba Lalo's user avatar
  • 57
1 vote
1 answer
14 views

To regularize every incidence of the term 'Heb' to "HEB" in pandas df

In a pandas df, trying to regularize every incidence of the term 'Heb' to "HEB" In a pandas df, trying to regularize every incidence of the term 'Heb' to "HEB" using sidetable, I ...
EmekaO's user avatar
  • 11
0 votes
0 answers
13 views

can't see all columns after implementing df.isna().sum() [duplicate]

I am working on a dataset on jupyter notebook that has 2000 rows and many columns. I wanna see how many missing items are in each columns Code: import pandas as pd fifa = pd.read_csv('fifa21_raw_data....
Rafsan Jahin's user avatar
-1 votes
2 answers
63 views

Stop tidyverse from renaming column in lapply

When I use group_by(col) inside lapply(), columns a and b are renamed to get(col). How do I stop this from happening? I would like to keep the original column names in the new objects. MRE Input df &...
myusername's user avatar
0 votes
1 answer
26 views

To find count of current employees in power bi using dax

i have a employee table like this. emp table My requirement is to count current employees in the orgnisation using dax. In the above table for employee id 1001 there were 3 rows of the data(i.e ...
Baktha Singh Babu's user avatar
0 votes
1 answer
59 views

Trying to complete R statements, getting Errors [closed]

I am working on an assignment and trying to create R statements, but keep getting errors. I have spent forever trying to figure it out. The below statements are what I have and have errors for almost ...
Simba's user avatar
  • 1
1 vote
1 answer
32 views

Auto Encoder didn't improve my neural network loss

so I'm trying to improve my neural network result by using auto encoder for preprocessing. The data looks more or less like this : Column A Column B Column C Column D Target -70 -76 -76 -80 1 -93 -...
Secondary Juggernaut's user avatar
0 votes
0 answers
15 views

In Pandas dataframe, I want to get only a part of the column [duplicate]

In Pandas Dataframe, I want to get only a part of the column and remove the rest and use it as a data cleaning technique. under Messages column the following are the list of items, I just want the A ...
Priya's user avatar
  • 1
0 votes
0 answers
16 views

Remove of unwanted characters in a column using pandas dataframe [duplicate]

I have a webscraped data that needs to be cleaned please For values that is divided by character "-" I want to keep the value before the character "-". For values like 5000000R (...
Donatus 's user avatar
0 votes
1 answer
40 views

How to get the matched groups in regex Python and save it as a new column

I have a dataframe and i want to find out, if there was any mentions of the firms that i'm looking for in DocumentIdentifier column. probably it should be done through Regex groups, but I'm not sure ...
Mostafa Bouzari's user avatar
0 votes
2 answers
43 views

R: Transpose Data from Wide to Long in Dataframe with many column names

I have a dataset that reflects responses to practice questions for a small group of people (~30). The dataset is very wide and has data for questions sent every day of the year. For each question that ...
Blair W's user avatar
0 votes
0 answers
15 views

How do I create a new dataframe after performing a calculation on an existing one? [duplicate]

I have a dataframe containing various customer details. It looks something like this: customer_id hours sales 1 5 20 2 7 10 3 2 16 1 4 ...
Dharmini's user avatar
-4 votes
0 answers
25 views

remove \n from each rows and a), b), c) , d) and the column type is pandas.core.series.Series [duplicate]

I have a column by name "options" and I have to clean it for future steps 0 a) qatar\\nb) brazil\\nc) germany\\nd) france 1 a) june - july\\nb) january - february\\nc) sept... 2 a) ...
chetan sharma's user avatar
0 votes
0 answers
19 views

How to fill nan values by comparing with other columns?

import pandas as pd import numpy as np df = pd.DataFrame({'SKU': [700, 701, 702, 702, 703, 704, 705, 705], 'CATEGORY': ['T', 'F', 'F', nan, 'W', 'W', 'T', nan]}) print (df) I sorted the original data ...
Shawnal's user avatar
0 votes
2 answers
45 views

How can I get a percentage breakdown of individual ethnicity variables from a question that allowed subjects to select multiple responses?

I ran this code to get the breakdown of ethnicities in my sample: dataset %>% group_by(ethnicity) %>% summarise(percent = 100 * n()/nrow(datset)) However, because subjects were able to ...
Louis 's user avatar
  • 11
3 votes
3 answers
68 views

Accumulate strings between years using midpoint assumption

I have a dataset of people-skills and the years they used those skills. I want to fill those skills between the years where they used them, because I assume they gained them midway between "use&...
Torin McFarland's user avatar
0 votes
1 answer
28 views

How to solve error related to name repair startegy with double pivot_longer

I have been trying to convert separately columns as long format, by using the following code: mtcars %>% pivot_longer(1:3, values_to = 'values') %>% pivot_longer(6:7, values_to = 'values1')...
12666727b9's user avatar
  • 1,067
0 votes
2 answers
59 views

Remove dot inside a number

I have the following JSON type file: [{ "sid": "s-mtl-1", "MaterialNumber": "0.4440", "MaterialWNumber": "W0.4440", "tubePrice": &...
MedvidekPu's user avatar
1 vote
1 answer
26 views

Finding common columns in two dataframes and retaining only those columns that are common between the two

I have a dataset that has gene names. The gene names In that dataset I wanted to extract them from two other datasets which are gd and cd. common_genes is a vector that has the the gene names that I ...
thole's user avatar
  • 117
0 votes
0 answers
12 views

Looking up a value and multiplying a column based on column name

First question so bear with me if I'm not asking well. I have a dataset that contains data in different units (ie ppb and ppm); I need to standardize all the data to the same units, but not all the ...
BeansMachines's user avatar
2 votes
3 answers
94 views

Extract number within brackets at the end of a string - (YYYY)

I am trying to create two separate columns in R. The issue I am running into is that the year does not separate from the previous column depending on the type of observation. Some names in the data ...
tangypancake4512's user avatar
0 votes
2 answers
52 views

Using Python to Remove a row in Excel based on Cell Value

I'm attempting to clean an excel file prior to sending it up to the database for calculations. By default when the Excel Report is exported out of our system (NextGen) it attaches a row that ...
jarodmwk's user avatar
0 votes
0 answers
40 views

can be a NLP used case to get number in a proper format in python?

I have an input -Dimension and I want my output in a specific form extracted from the text that I have in the dimension column. I have used df['output'] = df['Dimension'].str.extractall(r'(\d*\.?\d+)')...
Chayan Banerjee's user avatar
0 votes
1 answer
42 views

How to execute SQL scripts stored in cells of a temp table?

I have a fairly complicated dynamic SQL script that outputs dozens of scripts into a temp table. For example: UpdateTables AlterColumns UPDATE mydb.sch.tbl SET zipcode = null where col = ''; ALTER ...
manavjn's user avatar
  • 27
1 vote
2 answers
37 views

Delete all remaining observations from data frame when exceeding a specific value

I want to delete some observations from my data frame. For each individual (indicated by ID in the following), I have more than one observation and therefore more than one row in the data frame. The ...
unicorn's user avatar
  • 51
0 votes
0 answers
29 views

Joining two tables in R provides wrong output

I am trying to create panel data set by joining Table 1 and Table 2 Table 1 Table 2 The output I would like to receive is that I will have 4 columns: Powiat | Rok | Liczba bezrobotnych | Srednie ...
Viktor's user avatar
  • 11
0 votes
1 answer
52 views

Python code to remove all rows in a dataset that contain -1 anywhere in them

I have the following code for a dataset I am trying to clean. I need to remove several elements but currently removing any rows that contain -1 in them is causing some challanges. I have tried a ...
typod's user avatar
  • 49
0 votes
0 answers
32 views

how do i update row values based on latest df content using pandas or like a upsert function in pandas

I have multiple files that contains 12 weeks worth of sales data and i receive them every month. I will loop through the folder and append them into one df. Because the monthly file contains past 12 ...
ottogi's user avatar
  • 1
0 votes
2 answers
35 views

How can I not use nested lapply here?

Checking to see if date formats match within the date variables of my data frame using this code: df %>% select(date_vars) %>% lapply(function(x) x[!is.na(x)]) %>% lapply(...
Elias Wilson's user avatar
0 votes
1 answer
61 views

Compare two columns and replace the value in column 2 with NA if same as column 1

I have a data frame with 4 columns and I want to compare val1 and val2 and if the numerical values are the same then replace val2 with "NA".Notice row3 of the dataframe has 123 in val1 and ...
akang's user avatar
  • 599
0 votes
0 answers
30 views

accounting for returned items in sales data

I am working with medicine sales data and counting the dates on which patients have consumed the medicine. Patients can be on multiple medicines and I would have to count the number of days each ...
Zabi's user avatar
  • 1
0 votes
0 answers
59 views

How to delete the _MEI file in C:\Windows\Temp generated by running the exe program packaged by python

I run multiple python-encapsulated exe programs on the server. After the environment resources of each program are released, there are about 20 G, which causes the C drive to be full. I try to delete ...
starhu's user avatar
  • 1
2 votes
3 answers
80 views

Take unique values from certain columns and add them to an existing column

Example data: dat <- data.frame (meat = c("lamb, beef", "chicken, pork, beef", "venison, beef, lamb", "NA", "NA", "NA", "NA", &...
WaspWatcher's user avatar
1 vote
1 answer
37 views

How to merge two datasets in R by partially matching text in columns?

I apologize if this kind of question was already asked and solved. However, I tried to get to the solution without success. I have two datasets in R, DF1 and DF2; For example: DF1 <- data.frame(...
Strobila's user avatar
  • 317
2 votes
1 answer
34 views

How to automatically Filter off error tables using PowerQuery

How can I automatically filter off error tables using Power Query. I have this code that scans through my PDF files to extract the tables that I want in Excel. I want to automatically filter off a ...
chi005's user avatar
  • 73
1 vote
1 answer
45 views

function for reshaping data efficiently

I need to reshape a dataframe that looks like: holdc country_n popClass_n org P50M P50L P50U P95M P95L P95U P99M P99L P99U P99_9M P99_9L P99_9U 1 Austria Adults ADG 0.0445 ...
efz's user avatar
  • 435
-1 votes
2 answers
41 views

Identifying straight-lining responses in survey using R: i.e., respondents are listing "1" or "2" for all 11 questions in survey

In R, my dataset consists of 11055 obs. of 12 variables. The observations are grouped into groups of 11 for each individual denoted by their RID. As an example, the first few rows of my dataset look ...
Bob_123's user avatar
  • 19
1 vote
1 answer
32 views

I am trying to import a CSV file into MySQL workbench. There are some empty cells in a column in a non-empty row that I also want to import

The CSV file that I am importing has 566 rows. E.g., one of the columns called 'Safety Score' has 53 empty cells. I do not want to delete the rows entirely because there are some other useful columns ...
python_beginner's user avatar

1
2 3 4 5
69