Questions tagged [data-wrangling]
The data-wrangling tag has no usage guidance.
920
questions
0
votes
0
answers
32
views
Reshape data and expand cell into column and rows
I have dataset looking like:
The last column "Var3" contains multiple fields and rows in each cell, I want to expand each cell in "Var3" into rows and columns depends on the ...
0
votes
0
answers
15
views
How do I append columns from multiple rows to create one row by group? [duplicate]
I have a data frame that includes rows for each city, subject, and event - at which multiple measurements are recorded.
df <- data.frame(City = c("New York", "New York", "...
-2
votes
0
answers
24
views
Create new columns in a dataframe named after row values and filled with values from a different data frame [duplicate]
I have a data frame consisting of columns: locpix, res, year and CultivarID.
I would like to turn this data frame into one with locpix as the row names and the individual column names as combination ...
0
votes
2
answers
37
views
R: resample survey units at different proportions
I need to resample my survey. I have survey data from a country with 3 counties (1001, 1002, and 1003). The study surveyed 10 households in each county. I need to randomly pull the household id number ...
0
votes
0
answers
28
views
Generate 1000 sample based on a criteria [closed]
I have 3 numbers selected randomly from 3 different distribution (Exponential, Weibull, Normal) such that sum of all these numbers is always equal to 13.
Now I want to generate 1000 sample for such ...
0
votes
1
answer
45
views
How to color countries according to the count of rows on a map in R?
The columns of interest of the data look like this
wheels
name country
1 The Southern Star Australia
2 Wheel of Brisbane Australia
3 Wheel of Perth Australia
4 ...
-1
votes
1
answer
26
views
resampling dataframe/survey troubleshoot
I am having trouble resampling my dataframe. Need some help?
I have a household survey in country X. Country X is divided into 3000 counties of different population sizes. The % of sampled households ...
1
vote
1
answer
21
views
Rowwise mean imputation for groups of specified columns where >= 80% are non-NA values
I want to create a function that:
Takes the names of a group of columns (e.g. c("colA", "colB", "colC", "colD", "colE")) preferably using tidyselect ...
0
votes
1
answer
16
views
Counting frequencies from categorical data by group where data frame is in wide format
I have a patient dataset. Two groups of patients with several categorical variables include gender, medication and medication conditions (comorbidities). The example below is a fictitious dataset to ...
0
votes
2
answers
26
views
Converting unique values from data frame intro a reference Matrix
Hello lovely people of SO!
Guys I have the following raw dataset
ID_TRIAL<-c(1,1,1,2,3,4,5,5,5,6,6,6,7,7,8,8,8,8)
TYPE_FAIL<-c("A","B","C","F","A&...
1
vote
1
answer
18
views
Swap elements of a column name using gsub
In a previous post I was shown how to swap elements of a column name around for a specific instance...
Swap elements of variable names around
Is it possible to have a more general gsub function that ...
0
votes
2
answers
35
views
how to print paragraphs of strings out of columns in R?
Problem
I have a dataset df (see below) with several columns. And I would like to concatenate all columns and print a single paragraph row by row as you can see in Expected output. I would like to use ...
0
votes
2
answers
56
views
R: filter by multiple OR conditions
I need to filter a dataframe by multiple "OR" conditions. Let me explain.
I have a dataframe (total) with 1 million observations. One of the columns (id) contains id numbers ranging from 1 ...
3
votes
5
answers
65
views
Selecting number from string based on criteria
I have the following data set:
PATH = c("5-8-10-8-17-20",
"56-85-89-89-0-15-88-10",
"58-85-89-65-49-51")
INDX = c(18, 89, 50)
data.frame(PATH, INDX)
...
0
votes
1
answer
55
views
How can I replace nested for-loops to compute a new variable based on row-varying numbers of other datapoints?
I have a large dataset and need to reconstruct some information - specifically, for each agent (row), how many of its variable number of neighbours match it on a specific dimension.
Relevant data look ...
0
votes
1
answer
32
views
How to partition into equal sum subsets in R?
I have a dataset with a column, X1, of various values. I would like to order this dataset by the value of X1, and then partition into K number of equal sum subsets. How can this be accomplished in R? ...
-1
votes
0
answers
23
views
Insert conditional new observations basis another column in SAS dataset
The unprocessed dataset looks as below for link 1:
contains columns -
ID - unique key/identifier
Date - snapshot date of observation
Rating - Rating for the ID
Flag - flag to ...
1
vote
3
answers
36
views
Creating a table using two columns with multiple values in R
I have a sample dataset in R as follows:
ID <- c(1:10)
State <- c("AL", "AL", "AL", "TX", "TX", "TX", "TX", "TX", &...
1
vote
1
answer
17
views
How to combine similar rows of data into one in R?
I have a dataset as shown in the first table below. I would like to create a new dataframe that combines different versions of the same item (i.e. "googleV1" and "googleV2" are ...
0
votes
2
answers
27
views
Alternate input from 2 columns into one column in R
I have a data frame that is formatted like so:
GameId
colour
whiteElo
blackElo
1
white
2281
2293
1
black
2281
2293
4
white
2670
2595
4
black
2670
2595
6
white
2525
2470
6
black
2525
2470
and ...
1
vote
1
answer
23
views
How to apply the same functions to multiple datasets generated by different filter statements?
I have one df that will generate a few tibbles with the filter statement. Then I would like to apply a function, specifically rstatix::get_summary_stats to each tibble. I would also need to export ...
1
vote
2
answers
31
views
Add alternating string by group in R
I have a data frame formatted like so:
GameId
1
1
1
2
2
3
3
3
I want to create a new column called 'colour' which adds alternating rows with the colours white, black. I want this to reset ...
0
votes
1
answer
24
views
Taking value of previous row away from current row in R [duplicate]
I have a data frame formatted like so:
GameId
Eval
1
1.00
1
1.50
1
0.50
2
-1.00
2
2.00
2
1.00
3
1.50
3
-0.50
I would like a new column which accounts for differences between the current row ...
0
votes
0
answers
39
views
How do I combine groups with low number of observations into a newly defined group – for bar plot graph?
I have a dataset based on agronomic data of crop species. The data contains a count of the number of observations (popularity) of each crop species.
My goal is to construct a bar plot of the most ...
0
votes
1
answer
42
views
Adding Id number using group_by() in R [duplicate]
I have a data frame that's formatted like so:
evaluation
grouping
1.00
a
0.50
a
2.00
b
1.00
b
2.00
b
0.50
c
I want to create a new column with ID numbers for each grouping like so:
evaluation
...
0
votes
3
answers
103
views
Converting fixed length columns in bash
I have a text file generated using curl -s <URL> | lynx -stdin -dump -nomargins -width=9999
It looks something like this:
name1 yes yes no no
name2 yes yes 10 12 no yes
...
1
vote
1
answer
23
views
Swap elements of variable names around
I have variable names that follow the same pattern...
Measure name, item number, collection wave all separated by underscore.
For example, measure_1_baseline.
I want to swap the last two parts around ...
2
votes
2
answers
30
views
Creating a variable that appends together strings [duplicate]
I have a data frame like so:
moves
game_id
e2e4
1
cfc8
1
g6e4
1
f6g8
2
f8g5
2
e2e4
3
d8b6
3
h7a3
3
I want to create a column which appends each previous string in 'moves' to a list that ...
2
votes
2
answers
87
views
Reverse negatives for numeric values according to category in R
I have a data frame that looks like so:
colour
evaluation
white
1.00
black
- 0.50
white
- 0.60
black
1.50
Evaluations is a numeric data type.
I want to create code that flips all the symbols of ...
0
votes
1
answer
38
views
How to get presence/absence summary by month for each taxa to create ggplot bar plot?
I have fish stomach contents/diet data and I would like to get presence/absence information for each taxa in my df by month. Each observation (row) has information on the taxa absent (== 0) or present ...
1
vote
1
answer
33
views
Row bind then resave particular CSV files in a directory based on matching index column values in R
I am working with a large number of CSV files housed in the same directory. This time series data comes from sensors deployed at various locations over multiple time periods. Each file has a siteID ...
2
votes
1
answer
29
views
Selecting the group with the most observations in R
I'm trying to create a new column in my data frame that is the result of which of the three different columns has the highest value.
The data looks like this:
head(primary_game_fb, n = 20)
pitcher ...
0
votes
0
answers
18
views
How to export skimr::skim() results to a file with variable type reset on many data frames?
I have 2 data frames(more in real life). My goal is to generate summary reports with skimr package then export them as a file to a folder. They would also have different file names.
The problem that ...
0
votes
0
answers
16
views
How to allow skimr::skim_with function to output stats for date variable?
I have a date variable in my data frame, but skim(df) will output the date variable as character variable. How do I specify in the code that it needs to recognize the date, and output range of the ...
0
votes
3
answers
38
views
How to rename specific part of column names in R?
I have some column names as follows jan_20_5, feb_20_7, mar_20_3, apr_20_9, etc.
I have 12 such columns and would like to eliminate everything after the second "_" and append something ...
0
votes
0
answers
9
views
Extracting specific values from dataframe 1 into dataframe 2 [duplicate]
I have a data frame called lookup_table:
age.
mean.
sd.
2.
4.1.
0.7
3.
4.9.
0.3
4.
5.4.
0.8
And, I have another data frame df:
age1
bnt
2
4.1
2
3.9
3
4.5
4
4.8
4
5.5
4
5.6
I want to pull ...
1
vote
1
answer
20
views
How to use lapply to skimr:: skim multiple data frames then export them to Excel?
I have 2 data frames(more in real life). My goal is to apply the skim function then export them as excel to a folder. They would also have different Excel file names.
df1 <- data.frame(x = rep(3, 5)...
0
votes
1
answer
25
views
How can I join a variable from a different dataset with two if_else statements?
let's say I have df1:
Country
Date
Country A
2010-08-10
Country B
2010-10-10
Country C
2010-11-10
Country A
2011-09-10
Country A
2015-08-10
and I have df2
Ranking
Country
Date
5
Country A
...
0
votes
1
answer
29
views
How to append a column of values from each file in directory using another specific column as an index in Python
I have 10 tab-delimited files with this overall structure:
tracking_id
gene_id
gene_short_name
FPKM
ENSG00000121410.11
ENSG00000121410.11
A1BG
30.8275
Each file has the exact same number of rows ...
0
votes
0
answers
19
views
How to remove values lesser & greater than 1st& 99th percentiles after multiple imputation in R?
I want to remove data lesser& greater than 1st 99th percentiles.
I am using the following code
library(mice)
data(nhanes)
imp <- mice(nhanes)
I tried the following code but it does not work.
...
-1
votes
2
answers
17
views
Dataframe to a dictionary as some columns in a list as keys and one as value
I have a pandas dataframe df that looks like this:
col1 col2 col3
A X 1
B Y 2
C Z 3
I want to convert this into a dictionary with col1 and col2 in a list as ...
1
vote
1
answer
60
views
Creating a table grouping and Ordering elements based in criteria
Hello Everyone I hope you guys are having a great day!Hello lovely people of SO
I have the following dataset:
PRODUCT
REGION
REP
REG-YH-67
NORTH
JANE
REG-YH-68
NORTH
JANE
REG-YH-77
NORTH
JANE
REG-...
0
votes
2
answers
35
views
R: How to loop over a name-based selection of variables from a dataframe and for each create a new variable containing the column mean of the first?
I have a dataset containing a number of numeric variables whose names all start with "Ranking". For each of these variables, I want to add another variable to the dataset that contains the ...
1
vote
1
answer
33
views
Shift indicator value in column based on groups in second column
I have a dataset with many (arranged) groups of varying sizes and a second column that represents a systematic indicator for the selection of rows.
E.g. in the example below, starting from row 3 every ...
1
vote
2
answers
27
views
Creating a list with column-wise partitions of a data.frame
I have a data.frame with a single "identifier" column and many additional columns. I am interested in turning this data.frame into a list of length K, whose elements are sets of columns ...
5
votes
3
answers
52
views
New column based on values from other columns AND respecting pre-established rules
I'm looking for an algorithm to create a new column based on values from other columns AND respecting pre-established rules. Here's an example:
artificial data
df = data.frame(
col_1 = c('No','Yes'...
0
votes
0
answers
59
views
How to split text after a series of uppercase words and create a new row in Python
I used an OCR on thousands pdf files to create a CSV dataset of parliamentary speeches. In the dataset there are two columns: one contains the text of the speeches and the other contains the names of ...
2
votes
2
answers
57
views
Add in rows to matrices in a list so that all have uniform number of rows in the same order
I have many matrices that all have the same column names but different numbers of rows. I would like them all to be uniform length, and have the rows be ordered in consecutive increasing order by the ...
0
votes
1
answer
41
views
Sample values from a data.frame, run a function and use the results to create a third data.frame
I'm working with two data.frames, one contains parameter values ("params") and another contains covariate values ("data").
> head(params)
# A tibble: 6 × 4
`a[1]` `a[2]` `b[1]`...
1
vote
1
answer
26
views
How to read multiple files once using 'fread' in R [duplicate]
I need some help for reading multiple files (1.snplist,2.snplist, 3.snplist...) at once. Right now, I am using this,
library(data.table)
a1 <- fread('1.snplist')
a2 <- fread('2.snplist')
a3 <-...