7

Thanks for implementing shift in dt1.9.6 first. When I have many different groups, shift() is against expectations slower than my old code:

library(data.table)
library(microbenchmark)
set.seed(1)
mg <- data.table(expand.grid(year = 2012:2016, id = 1:1000),
                 value = rnorm(5000))
microbenchmark(dt194 = mg[, l1 := c(value[-1], NA), by = .(id)],
           dt196 = mg[, l2 := shift(value, n = 1,
                               type = "lead"), by = .(id)])
## Unit: milliseconds
##   expr      min        lq      mean    median       uq        max  eval
##  dt194  4.93735  5.236034  5.718654  5.623736  5.74395   9.555922   100
##  dt196 83.92612 87.530404 91.700317 90.953947 91.43783 257.473242   100

A detailed script is here: https://github.com/nachti/datatable_test/blob/master/leadtest.R

Did I misapply shift()?

Edit: Avoiding := doesn't help (@MichaelChirico):

microbenchmark(dt194 = mg[, c(value[-1], NA), by = id],
               dt196 = mg[, shift(value, n = 1,
                                   type = "lead"), by = id])

## Unit: milliseconds
##   expr       min        lq     mean    median        uq       max neval
##  dt194  5.161973  5.429927  5.78047  5.698263  5.798132  10.42217   100
##  dt196 79.526981 87.914502 92.18144 91.240949 91.896799 266.04031   100

Apart from this using := is part of the task ...

8
  • 2
    I do not believe it is a reasonable expaction that shift can beat c(value[-1], NA). There might be potential for improvement, but the base R solution here is extremely optimized and a function that offers much more versatility (e.g., non-vector input) must be slower.
    – Roland
    Feb 3, 2016 at 14:56
  • 1
    @Roland: It's just slower for many groups. If there are just a few groups (e.g. 2), shift() is slightly faster. For an example click the link above.
    – nachti
    Feb 3, 2016 at 15:04
  • 1
    I think you should take this to the data.table bug tracker.
    – Roland
    Feb 3, 2016 at 15:23
  • 1
    Ok. Reported it here: github.com/Rdatatable/data.table/issues/1534
    – nachti
    Feb 11, 2016 at 15:27
  • 1
    You shouldn't be using `` := `` in your benchmarks. Just run mg[ , c(value[-1L], NA), by = id] and mg[ , shift(value, 1, "lead"), id] Feb 11, 2016 at 16:00

1 Answer 1

2

In data.table version 1.14.3 this has been resolved and shift becomes faster than ever.

library(data.table)
library(microbenchmark)
set.seed(1)
mg = data.table(expand.grid(year=2012:2016, id=1:1000),
                value=rnorm(5000))
microbenchmark(v1.9.4  = mg[, c(value[-1], NA), by=id],
               v1.9.6  = mg[, shift_no_opt(value, n=1, type="lead"), by=id],
               v1.14.3 = mg[, shift(value, n=1, type="lead"), by=id],
               unit="ms")
# Unit: milliseconds
#     expr     min      lq    mean  median      uq    max neval
#   v1.9.4  3.6600  3.8250  4.4930  4.1720  4.9490 11.700   100
#   v1.9.6 18.5400 19.1800 21.5100 20.6900 23.4200 29.040   100
#  v1.14.3  0.4826  0.5586  0.6586  0.6329  0.7348  1.318   100
1
  • Thanks for the implementation!
    – nachti
    Nov 23, 2021 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.