Questions tagged [data.table]
The R data.table package is an extension of data.frame built for fast in-memory data analysis. Use the dt tag for the DataTables package with Shiny (DT).
12,640
questions
0
votes
1
answer
29
views
R data.table way to create summary statistics table with self-defined function
I am in the process of converting to data.table and so far have not been able to find a data.table way to create a table with summary statistics based on a self-defined function. Until now, I have ...
0
votes
1
answer
19
views
Is there a way to aggregate multiple subgroups within a larger group/column with data.table?
New R user here and I am trying to aggregate multiple groups of data within a larger group, for example Males/Females of Adults by Census Tracts. Currently I am writing it as:
DEmale <- ...
1
vote
2
answers
30
views
Directly (re)naming summarized columns with data.table
I want to check the proportions of certain categories inside column of a data.table with respect to some grouping variable. A working chunk of code is attached that does just this:
require(data.table)
...
1
vote
1
answer
44
views
data.table: force <NA> response
In my example:
library(data.table)
id <- 1:10
x <- rnorm(n=10,mean=10)
groups <- ifelse(x<5,"D","S")
df <- data.frame(id,x,groups)
str(df)
# 'data.frame': 10 obs. of ...
0
votes
2
answers
54
views
Conditional Number Formatting in R
Can anyone tell me how I can conditionally format my data.table so that numbers >= 1000 do not have decimal places and those below 1000 only have one decimal place?
Here's an image of my data.table
2
votes
2
answers
33
views
Count unique values before a row itself for each group in data.table
I have a data.table like this;
df <- data.table(Date = c(seq.Date(from = as.Date('2022-01-01'),to = as.Date('2022-01-07'),by=1),
seq.Date(from = as.Date('2022-01-01'),to = ...
0
votes
2
answers
36
views
Dynamically calling variables within a function for a function using formula notation (in this case xtabs)
I have data as follows:
dat <- data.table(type=c("V","X","Y","Z","V","X","Y","Z"), cat=c("A", "B",...
1
vote
0
answers
58
views
Function with several complex rules
In my case:
# Open dataset
library(dplyr)
my.ds <- read.csv("https://raw.githubusercontent.com/Leprechault/trash/main/test_ants.csv")
my.ds
# days nest AT
# 1 0 2 10.9200
# ...
1
vote
1
answer
31
views
data table conditional cross join with two conditions
I am trying to cross join two data tables, but with two conditions that I need to specify before the join, because if I do not, the memory limit kills it. (See this annoying feature of data table.)
...
1
vote
1
answer
30
views
Join by time stamp within time period
I have two tables, which I would like to join by date. If an instance falls within a color period, I would like that instance to receive said color as a variable (see joined):
instance <- read....
1
vote
1
answer
15
views
r - Need a workaround for copying a data.table in a reference class
I'm trying to copy a data table within a reference class but I'm unable to do so. Here is a small example
library(data.table)
Example <- setRefClass("Example",
fields =...
2
votes
2
answers
37
views
Subset data.table containing groups with at least 2 non-NA values
In the data.table below, I want to subset only the groups (in
column x), which have at-least 2 non-NA values.
dt <- data.table(x = c(rep("a", 5), rep("b", 3), rep("c", ...
2
votes
1
answer
15
views
Looking up data in another data.table from j
I have been running into a computation time issue when working with large datasets and applying functions that use multiple columns of the data.table as inputs. Here is a simplified example:
require(...
1
vote
1
answer
43
views
"Temporarily" filtering dataset in R to create a new variable?
I am trying to capture the date of the laboratory assay that falls closest to:
30 days after a person was vaccinated (d3m1)
90 days after the date of their vaccination (d3m3), and
180 days after the ...
1
vote
0
answers
72
views
Error for cross join / cartesian merge in data table, any solution?
I am trying to do a conditional cross join in data table, and I am running into this error:
Error in vecseq(f__, len__, if (allow.cartesian || notjoin || !anyDuplicated(f__, :
Join results in more ...
0
votes
0
answers
29
views
Smooth the y-axis and then extrapolate using R
In the data.table below, I want to smooth the values on y-axis, which are not aligned to the x-y curve, and then extrapolate to find the missing values.
E.g. in group P1, value y = 0.850 at x = 77.5 ...
0
votes
0
answers
55
views
How to rename columns in a list of data tables according to a rule?
I would like some help with the following, have searched around for answers on here (e.g. Changing Column Names in a List of Data Frames in R) but cannot find an answer.
I have a list of data tables ...
1
vote
1
answer
55
views
Using rlang double curly braces {{ in data.table
Problem
The {{}} operator from the rlang package makes it incredibly easy to pass column names as function arguments (aka Quasiquotation). I understand rlang is intended to work with tidyverse, but is ...
1
vote
2
answers
59
views
Fill the missing quarter values from annual values using R
In the data.table below, column value shows the sales growth at each quarterly and annual event of groups A and B for the years 2017 and 2018.
Sometimes, groups A or B don't hold the quarterly event ...
2
votes
1
answer
47
views
Manipulation of Variables with Similar Names in data.table
I was wondering if there was a more terse way of manipulating variables (in my context dividing) variables that have near identical names.
Please consider the following dataset:
library(data.table)
n =...
0
votes
4
answers
52
views
Finding equal rows between dataframes in R
I have the following data set as example:
df1 <- data.frame(V1 = 1:10, V2 = 1:10, V3 = 1:10)
df2 <- data.frame(V1 = 5:1, V2 = 5:1, v3 = c(1, 4, 5, 2, 3))
If a row in df1 are present in df2, I ...
0
votes
1
answer
18
views
data.table melt 3 columns into one
After merging several data.tables, I get one that looks somewhat like this:
dt<- data.table(age = rep(c("5-16", "17-39"), each = 3), dur = rep(c("short","medium&...
1
vote
0
answers
50
views
Differences in numeric representation between read.csv and data.table::fread
I noticed some strange behaviour reading csvs into R using data.table::fread() (version 1.14.2) and read.csv(), where certain numeric values are not identical across the two read methods. A minimal ...
1
vote
3
answers
112
views
Fastest way to check to which bracket a value belongs for large data
The data below has lists in the strata column. I would like to use the list in the strata column as the cut-off values. The solution provided below comes from r2evans.
The problem is that I am using ...
3
votes
3
answers
53
views
Return first matched row using date lookup to a date range lookup table
I have 2 dataframes, one with list of dates (df1) and another one with date ranges by ID (df2). I would like to add a column from df2 if a date in df1 sits within the date range in df2.
## This is my ...
1
vote
1
answer
25
views
Find common values of two lists in R data table
I have a data table test:
id
array1
array2
1
c(1, 2, 3, 4, 5)
c(3, 4, 5)
2
c(6, 7, 8, 9, 10)
c(6, 7, 0)
> str(test)
Classes ‘data.table’ and 'data.frame': 2 obs. of 3 variables:
$ id : ...
2
votes
2
answers
35
views
create grouped "order" variable in data.table in R
I have a data.table:
example <- data.table(year = c(2016, 2017, 2018, 2016, 2017, 2020),
ID = c("A","A","A", "B", "B","B"))
...
2
votes
2
answers
36
views
R: cross join column on itself excluding symmetric results
I have a data table init that looks like this:
> init
+---+
| id|
+---+
| a|
+---+
| b|
+---+
| c|
+---+
I want to obtain all pairs for id column, so I need to cross join the init data ...
2
votes
1
answer
35
views
Aggregate character string into vector in R
I have a data table test:
id
key
1
2365
1
2365
1
3709
2
6734
2
1908
2
4523
I want to aggregate unique key values by id into vector using data.table package.
Expected output:
id
key_array
1
&...
0
votes
0
answers
51
views
Why wont my function return the data.table?
I have a spending distribution function I created. If I run the code line by line it will display the correct output but when using the function it will not display anything. I've tried print instead ...
0
votes
2
answers
55
views
Creating column that takes mean of X for specific group id when date1 < date2
in (R)
for an event study I'm trying to create a column that calculates the mean of ccu_avg for a specific combination of appid and Eventdate1. One appid has multiple events so it has to be divided by ...
1
vote
1
answer
41
views
Reading TXT files in zipped folder with fread without unzipping in r
I am trying to read TXT files in a compressed folder with two folders inside.
Data (main folder)
river (subfolder)
treatment (subfolder)
Inside river I have many files in txt format and I am using ...
0
votes
0
answers
30
views
Manipulating a variable in one object also manipulates another object in R [duplicate]
I am creating a copy of an object, and then when I manipulate the copy, also the original is changed. Is anyone familiar with this?
Reproducible example -
a <- data.table(av = c(1, 2, 3, 4, 5), bv =...
0
votes
2
answers
100
views
R data.table: How to calculate ratio of sum of values, for columns from vector, within the group, vs. the rest of the values in the columns in table?
Let say we have a DT:
A
B
C
1
1
1
1
2
3
2
3
5
2
4
7
3
5
9
3
6
11
Let's say we want to obtain the ratio of mean value of B and C within each group A c(1,2,3) to mean values of B and C but for ...
0
votes
1
answer
40
views
How can I create lag data (all forward combinations) with data.table in R?
I need to create a dataframe with all possible combinations of a variable. I found an example using data.table that works like this:
df <- data.frame("Age"=1:10)
df <- setDT(df)...
1
vote
1
answer
28
views
Constructing Momentum Factor in R Studio - Fama & French
I want to construct a momentum factor which is part of Fama and French Regression. We have a monthly data table starting from 1980-12-31 to 2021-12-31, and a time series of stock returns, given as a ...
0
votes
2
answers
63
views
What is the correct syntax when calling a variable name from within a function
I have data as follows:
library(data.table)
set.seed(1)
year = c(rep(2000,5), rep(2001,5), rep(2002,5), rep(2003,5), rep(2004,5))
DT <- data.table(panelID = sample(10,10), ...
3
votes
1
answer
38
views
How to summarize by ID from two different data frames using data.table
I have multiple data sets that I have been combining into one summary data frame in R.
I am at the last step, but I am stymied.
Here is a dummy scenario: Imagine the two data frames df1 and df2 being
...
0
votes
5
answers
47
views
In R, get a frequency distribution of IDs by count of distinct / unique categorical (factor) attribute
Background
I've got a dataframe df:
df <- data.frame(ID = c("a","a","a", "b", "c","c","c","c","d","d&...
0
votes
2
answers
33
views
In R, conditionally count IDs who meet an ANY condition on a certain attribute
Background
I've got a dataframe df:
> df <- data.frame(ID = c("a","a","a", "b", "c","c","c","c","d",&...
0
votes
1
answer
29
views
How to combine columns into one column ignoring NAs (data.table)
I have a data.table like below. I want to merge two columns A and B into one but excluding the missing values (if exists)
dt <- data.table("title"=c("A", "B", NA, &...
0
votes
1
answer
30
views
na.locf in data.table conditional on contents of a row
I have a data.table where I want to carry forward the last observation. A sample from my data table looks as follows:
V1 V2 V3 V4 V5 V6
1 Bond 1 Bond 1 Bond 1 Bond 2 Bond 2 ...
2
votes
0
answers
38
views
Why data.table::setnames acts on multiple dataframes while I'm specifying just one? [duplicate]
As you can read in the title I'm wondering why data.table::setnames acts on multiple dataframes while I'm specifying just one. See attached code below:
library(data.table)
# create dataframe
df <- ...
1
vote
1
answer
143
views
Sum in R based on a date range and another condition?
I am working on a dataframe of baseball data called mlb_team_logs. A random sample lies below.
Date Team season AB PA H X1B X2B X3B HR R RBI BB IBB SO HBP SF SH GDP
1 2015-04-06 ARI ...
0
votes
1
answer
34
views
How to remove a specific character from a data.table
I have a data.table like below.I would like to remove all \$ from the data.table. This character may exist in any column.
dt <- data.table("title"=c("First $Title", "Second ...
1
vote
2
answers
38
views
R jsonlite does not parse JSON as per keys correctly
When converting json to columns in R using jsonlite, I came across a bug where the column values were in the wrong columns. I tried replicating it -
R> dt <- data.table(id = 1:2,
json = c('{&...
0
votes
2
answers
38
views
How to apply data table .I function to only specific elements in data table?
Suppose we generate the following data table by running the simple data.table code shown immediately beneath, where the code numbers each element in the data table (Nbr column) and further letter-...
1
vote
3
answers
75
views
Replace values in one data.table from another data.table [duplicate]
I have a data.table called big in which I want to replace all corresponding values from data.table called new_big.
library(data.table)
big <- structure(list(id = c("B", "C", &...
0
votes
1
answer
46
views
R data.table non-equi join based on "not equal"
I would like to merge the data.table to itself based on values where id's are not equal.
Here is a small example:
library(data.table)
#Two tables:
dt_1 <- data.table(id = c(1,2,3),x = c(2,3,4))
...
0
votes
1
answer
24
views
How to apply ConvertColor function to RGB values listed in a data.table in R
I'm running an experiment where I need to compare the consistency of RGB values selected from two different time points. To do this, I need to convert RGB to XYZ values which I know I can do using ...