1

I would like to use data.table functionality to run monthly regressions and return coef, residuals, etc.

In the example code below, I want to be able to see each named list. My first reg.list call does that, but it requires me to call my fit.lm helper function for each list I want to return. Which is probably not efficient. My second reg.list call only calls it once, but I get all my variables strung together instead of in named lists. Maybe I am not approaching this the correct way.


f1 <- data.table(datadate = '1/1/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
f2 <- data.table(datadate = '1/2/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
f3 <- data.table(datadate = '1/3/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
fdata <- rbind(f1,f2,f3)

fit.lm <- function(mdate) {

  cols <- c("datadate", "id", 'y', 'x')
  load <- fdata[datadate == mdate, ..cols]
  mod <- lm(formula = 'y ~ x', data = load)
  return(list(coef = list(mod$coefficients), residuals = list(mod$residuals), r2 = list(summary(mod)$r.squared)))
}

reg.list <- fdata[, list( coef = fit.lm(datadate)$coef,
                          residuals = fit.lm(datadate)$residuals,
                          r2 = fit.lm(datadate)$r2), by = datadate]


reg.list <- fdata[, list(mod = fit.lm(datadate)), by = datadate]
reg.list[[2]]

In my second call to reg.list, I would have to access the output as follows:

reg.list[[2]][1]
reg.list[[2]][2]
reg.list[[2]][3]
reg.list[[2]][4]
reg.list[[2]][5]
reg.list[[2]][6]
reg.list[[2]][7]
reg.list[[2]][8]
reg.list[[2]][9]

That doesn't seem like a very user friendly way. I think I am doing something wrong. I would like to do something like:

reg.list[2][['coef']]
reg.list[2][['residuals']]
reg.list[2][['r2']]
1
  • 1
    You can remove the list part, since it already is a list, and write fdata[, fit.lm(datadate), by = datadate]. Then you have coef, etc. as column names and can access them the usual way. Sep 18, 2019 at 16:44

2 Answers 2

0

Building on what @IceCreamToucan is saying, I think the main issue here is that the fit.lm function is pushing out lists of different lengths, so data.table isn't able to flatten the result. If you make your function push out the same number of values for each item of the return list, data.table can flatten the result and keep your data tidy. see code below ->

f1 <- data.table(datadate = '1/1/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
f2 <- data.table(datadate = '1/2/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
f3 <- data.table(datadate = '1/3/2019', id=paste('id', 1:100, sep=''), x=runif(100), y=runif(100))
fdata <- rbind(f1,f2,f3)

fit.lm <- function(mdate) {

  cols <- c("datadate", "id", 'y', 'x')
  load <- fdata[datadate == mdate, ..cols]
  mod <- lm(formula = 'y ~ x', data = load)

  return(c(as.list(mod$coefficients),
           'sse' = sqrt(mean(mod$residuals^2)), 
           'r2' = summary(mod)$r.squared
           )
         )
}



fdata[,fit.lm(datadate), by = datadate]
0
0

Thanks to both @IceCreamToucan and @Bryan for your helpful information. Using what I learned from both of you, I was able to build what I wanted (see code below). I wanted to return 3 lists (or objects) so that coef would be a list of however many coefs the regression had and residuals would be a list of however many residuals.


fit.lm <- function(mdate) {

  cols <- c("datadate", "id", 'y', 'x')
  load <- fdata[datadate == mdate, ..cols]
  mod <- lm(formula = 'y ~ x', data = load)

  return(list('coef' = list(mod$coefficients),
    'resid' = list(mod$residuals), 
    'r2' = summary(mod)$r.squared))

}
reg.list <- fdata[, fit.lm(datadate), by = datadate]


This allows me to easily see my coeffients like so...


coefs <- do.call("rbind", reg.list$coef)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.