3,571
questions
0
votes
2
answers
35
views
How to combine the first row and second row into a new row in a new column in Power Query?
I need help in cleaning up data in Power Query. I'm trying to make another column based on the description column. How can I create a new column that will combine the current row and the next row as ...
0
votes
0
answers
8
views
How to demean and decor relate a set of univariate data samples?
Suppose there are a set of samples drawn from a Gaussian with mean 10 and variance 2 as
x = 10 + chol(5) * randn(1, 100)
How do I demean the data with 0 and decorrelate with 1 so that a new set of ...
0
votes
0
answers
29
views
How do I delete all rows that have non-integer values in a panda dataframe? [duplicate]
I need to clean up a data frame, in which I have to delete all rows that have non-integer values. There's only one columns that have this issue.
My code right now:
data=pd.read_csv('house_data_2024....
0
votes
0
answers
14
views
Difference between Changelo&documenting the cleaning data process
Is there a difference between: Changelog and documenting the data cleaning process in data process cleaning?
I was learning this on the coursera google certificate and I wanted to know if it's the ...
0
votes
2
answers
40
views
how to filter keywords from a list of keywords from the job description in excel only
I have 100 000 data. from there i gather some list of keywords. i want to find out keywords mentioned in a job description from that list of keywords in excel.
I tried to use that formula given below ...
1
vote
1
answer
34
views
How can I replace multiple rows in a polars dataframe with a dict mapping?
So lets say I have an excel sheet/csv of users and their salaries records. I have created an account for each user in a db and want to create salary record for each using the ids from the db.
import ...
2
votes
2
answers
49
views
Is there a better way to replace all non-ASCII characters from specific columns on a DataFrame?
There are some sentences and words in Chinese and Japanese that I just want to drop.
Or if there is a better solution than just dropping them, I would like to explore them as well.
import pandas as pd
...
-5
votes
1
answer
35
views
How to remove duplicate rows in postgresql
I want to delete the duplicate rows from a table nashvillehousing. But when I try to use a CTE and delete the duplicate rows it isnt working and I get "relation cte doesnot exist",SQL state: ...
0
votes
1
answer
49
views
Postgres: Combining rows which are identical in all columns but one
(Table where e.g. the 2nd and 3rd columns are identical except for the Consoles column)
I have a table of video games which has 6 columns: Game_Title, Consoles, ESRB_Rating, Content_Descriptors, ...
1
vote
1
answer
37
views
Different Data Name Output [duplicate]
I want to count the highest age of diabetes in this dataframe. Where the expected output of this code is like this:
age
25 14
31 13
41 13
29 13
43 11
22 11
28 10
33 10
38 10
...
0
votes
1
answer
36
views
Calculating net assets with missing values: discrepancy in results
I'm working with a wealth component dataset that includes variables for housing, business, financial assets, loans, and non-housing loans. These variables have varying levels of randomly allocated ...
0
votes
0
answers
9
views
How can I do text pre-processing in IBM SPSS Modeler?
I want to do the standard text cleaning of removing stop word, stemming, tokenisation, etc. I think SPSS has limitation but how do people get around with this using SPSS?
Chatgpt said to use python ...
2
votes
2
answers
65
views
How to combine two columns into `{key:value}` pairs in polars?
I'm working with a Polars DataFrame, and I want to combine two columns into a dictionary format, where the values from one column become the keys and the values from the other column become the ...
3
votes
3
answers
72
views
How can I replace null values in polars with a prefix with ascending numbers?
I am trying to replace null values in my dataframe column by a prefix and ascending numbers(to make each unique).ie
df = pl.from_repr("""
┌──────────────┬──────────────┐
│ name ┆...
0
votes
0
answers
29
views
missing_values in numpy.genfromtext is not working
i realized numpy.genfromtext's filling_values only trigger when there is a miss-match for types,
for example here:
i tried both with version numpy==2.0.1 and numpy==1.26.3
import numpy as np
import io
...
0
votes
0
answers
27
views
TypeError: Could not convert string to numeric during groupby mean operation in Pandas
I'm working on a data analysis project using Pandas to analyze housing prices in London boroughs. I'm trying to group the data by borough and year and then calculate the mean of the average housing ...
0
votes
2
answers
41
views
dynamic approach to identify and standardize similar names automatically in pandas or data cleaning
I have a DataFrame with a column of publisher names that contains various minor variations of the same publisher. For example, entries such as "Harlequin Romance", "Harlequin Blaze"...
1
vote
2
answers
108
views
Expression.Error - We cannot convert a value of type Function to type List
This M (Power Query) function gives this error "[Expression.Error] We cannot convert a value of type Function to type List." The objective is to output a table column being the difference ...
0
votes
1
answer
52
views
How to get the Friday date from a week start date
In Excel, given a date like 3/17/2024, how can I get the date corresponding to Friday in the same week?
3
votes
1
answer
66
views
Ignoring error in Table.ExpandTableColumn or auto-replace with null
I am loading 100s of XML files from a folder and Power Query has created a function from the first file I had loaded. The problem is not all the XML files have a similar structure. Some have nodes ...
1
vote
1
answer
39
views
Power Query - Add column with path for each file
I have two queries (working) hat do exactly what I want:
they scan a folder for DBF files with a particular name
convert each one into a table
combine the tables into one long table
I want to add a ...
2
votes
3
answers
102
views
Optimizing pandas performance on large datasets
I'm working with a large dataset (~10 million rows and 50 columns) in pandas and experiencing significant performance issues during data manipulation and analysis. The operations include filtering, ...
0
votes
2
answers
39
views
Pad column numbers while using separate_wider_delim in R
I have a dataset in R that contains a column that contains a string that I want to separate into multiple columns using separate_wider_delim from the tidyr package.
What I want to do is to pad the ...
1
vote
2
answers
50
views
Identifying the max in a group using PowerQuery
I have a table:
trip_id
stop_sequence
Y1
1
Y1
2
Y1
3
Y1
4
Y1
5
Y2
1
Y2
2
Y2
3
Y2
4
I want to add a custom column that has a value of TRUE for the record which has the highest value of ...
-1
votes
1
answer
42
views
How can I scrape data efficiently and clean it? [closed]
I scraped data from a website and I am having trouble cleaning
this is the code, I scraped the data with. Is that best practice?
import requests
from bs4 import BeautifulSoup
import json
...
0
votes
1
answer
112
views
How to remove small tilde and other special characters from excel? [closed]
I am learning excel from scratch, and I have a dataset of around 7000 entries. Some of these entries have texts such as:
This obviously is not clean data, and I wish to highlight it. I tried using ...
0
votes
1
answer
71
views
checking the presence of any of the character from a certain set of characters in an Excel cell, and returning values accordingly
I have a data set in which there are several cells that should contain numbers but are formatted as text.
What I am trying to do is enter a formula that checks for the presence of any of the following ...
1
vote
1
answer
63
views
How do I split addresses with no commas, and irregular layouts in a Pandas data frame?
I have a column in a data frame like this (it includes many more address):
Address
287 Andover Pl Robbinsville NJ 08691
1 Oxford Ct Princeton Jct NJ 08550
244 N Post Rd Princeton Jct NJ 08550
3 ...
1
vote
3
answers
95
views
Unstacking table with multiple columns in Power Query
I am trying to convert this table:
ID rating cntry Index
Albert 603 pl 0
Peter 912 at 1
Peter 907 at 2
Albert 608 pl 3
Franz 833 pl ...
2
votes
2
answers
173
views
How do I remove quotation marks from characters strings in Power BI?
I have unstructured data with quotation marks I'd like to remove using Power Query. Here's an example:
Have:
chrome has been “processing” for 8 mins now.
Want:
chrome has been processing for 8 mins ...
0
votes
1
answer
90
views
How do I update timestamp values in a BigQuery table?
I need to update some date values that were stored incorrectly. I have two tables of data that I am combining. Basically, table1 and table2 contain consecutive months of the same data. Table1 has the ...
2
votes
1
answer
80
views
Tidying data frame with multiple variables within multiple columns
I'm working with a "molten" data frame, though perhaps "volcano" is a more apt description of it.
Basically, there are two columns in this data frame that break the first rule of ...
-2
votes
1
answer
32
views
I've got a long list of data for items purchased by customers, but I want to group the data into categories. How do I do it? [closed]
It is a very long list of mixed items and I want to categorise them. How can I do it on excel other than manually categorising them which is near impossible
Tried conditional statements but i dont ...
0
votes
1
answer
43
views
How to get a cleaned dataframe after using ydata-profiling on it?
I know it won't probably be perfectly cleaned (since it it an automated task) but I would like to understand how to get the cleaned dataframe (not the original) after the original has been processed ...
-2
votes
1
answer
35
views
How to incorporate data cleansing into trained model
If I cleanse the data and impute median value into NaN values, am I supposed to somehow incorporate this into my model that will be used on the test data? In other words, doesn't my test data need to ...
0
votes
0
answers
31
views
Regex that stop matching till new line [duplicate]
I want regex that match words 'terimakasih' and stop when encounter comma,period or newline
here's my regex when i try it :
\bterimakasih\b[^,.]*[\n,.]
but when i try it in this text :
halo zulhelmi, ...
0
votes
0
answers
38
views
Dates changing format when running excel macro
I have this excel macro to clean excel files and remove hard spacing and non printable characters, but nothing to do with changing the dates format or anything at all but when its saved as a unicode ...
0
votes
3
answers
65
views
Splitting multiline text before data cleaning.What option for splitting Oracle userform data?
I'm a medical doctor, just starting Python, so excuse me if I misuse some terms.
We use an Oracle Userform for our Electronic Health System.
Blood Pressure data appears in a text field, and can ...
0
votes
1
answer
67
views
How can I clean up this data that was converted from JSON to a Pandas DataFrame?
I have a JSON list that I normalized with the following code:
json_nor = pd.json_normalize(data, max_level=1)
pd.set_option('display.max_colwidth', None)
#view json_nor
json_nor
This is the result ...
0
votes
0
answers
16
views
Formatting phones numbers in pandas [duplicate]
Someone please tell me why the perfectly good code below is not taking out the non-alphanumeric characters from these phone numbers..? As you can see form the code below, when I print out the column, ...
2
votes
6
answers
98
views
R function for assigning value to grouped data based on a single row condition
I would like to create a new column of data based on a value within a grouped variable.
For instance, given the following dataset, I would like to assign all rows of a group to have the value TRUE if ...
0
votes
0
answers
19
views
Recoding Multiple Likert-scale Columns at Once
I usually do this the hard way, but I'm sure one of you coding experts has something less tedious.
Using this data set below:
#Example Dataset
Q1 <- c("Agree", "Disagree", &...
2
votes
1
answer
113
views
How to clean a messy data in a column in csv file with python or Power BI
I am working on a dashboard to show employees (technicians on the field) productive working hours.
I have a CSV file exported from the app that the technicians use when they are executing their tasks. ...
0
votes
1
answer
36
views
How to count all instances of two different entries that should've been one?
I apologize if this is a silly question, but I've recently started to learn SQL and I'm currently trying to clean a dataset.
However, some entries have mistakes like having a ' ' after the text (I ...
0
votes
2
answers
65
views
R aggregate() and distinct() functions only cleaning some of my data
I am currently struggling with trying to impute or remove rows in R that are mostly duplicates...except for three columns. I am working with a mortality dataset where I have countries/age groups/...
-1
votes
1
answer
32
views
Remove duplicate (almost) rows based on value in the distinct column
Consider this result in SQL Server:
ID Check
-----------
9052 N
9052 Y
2049 N
2049 Y
6940 N
6940 Y
7941 N
8118 N
8187 N
How can I delete the duplicate ID rows and leave ...
0
votes
1
answer
55
views
How do I convert this string 'April 9, 2013' into the format 'dd-mm-yyyy' in MySQL
I have been trying to convert a column consisting of dates in the format 'April 9, 2013' into the format '%d/%m/%Y' which would result in '09-04-2013'.
I have tried using the STR_TO_DATE function and ...
-1
votes
1
answer
48
views
PowerBI - How to pivot answer lists to column answers
In PowerBI, I have a data table which includes a question column and an associated answer column.
In the answer column there are a mixture of lists (lists may contain one or more answers) and non-list ...
0
votes
1
answer
49
views
How to loop over named list in R?
I am simulating a model for different values of the parameters. Essentially I obtain a list object results containing several dataframes.
library(MASS)
model_beta <- 0
model_se <- 0
model_p <...
0
votes
0
answers
46
views
Unable to clean a string column with pyspark
I'm currently connecting to an ibm db2 database, reading a table into a pyspark dataframe and writing it into hdfs as parquet. Out of around 200 tables around 10% are throwing out the same error:
...