Thanks for implementing shift in dt1.9.6 first.
When I have many different groups, shift()
is against expectations slower than my old code:
library(data.table)
library(microbenchmark)
set.seed(1)
mg <- data.table(expand.grid(year = 2012:2016, id = 1:1000),
value = rnorm(5000))
microbenchmark(dt194 = mg[, l1 := c(value[-1], NA), by = .(id)],
dt196 = mg[, l2 := shift(value, n = 1,
type = "lead"), by = .(id)])
## Unit: milliseconds
## expr min lq mean median uq max eval
## dt194 4.93735 5.236034 5.718654 5.623736 5.74395 9.555922 100
## dt196 83.92612 87.530404 91.700317 90.953947 91.43783 257.473242 100
A detailed script is here: https://github.com/nachti/datatable_test/blob/master/leadtest.R
Did I misapply shift()
?
Edit: Avoiding :=
doesn't help (@MichaelChirico):
microbenchmark(dt194 = mg[, c(value[-1], NA), by = id],
dt196 = mg[, shift(value, n = 1,
type = "lead"), by = id])
## Unit: milliseconds
## expr min lq mean median uq max neval
## dt194 5.161973 5.429927 5.78047 5.698263 5.798132 10.42217 100
## dt196 79.526981 87.914502 92.18144 91.240949 91.896799 266.04031 100
Apart from this using :=
is part of the task ...
shift
can beatc(value[-1], NA)
. There might be potential for improvement, but the base R solution here is extremely optimized and a function that offers much more versatility (e.g., non-vector input) must be slower.shift()
is slightly faster. For an example click the link above.:=
`` in your benchmarks. Just runmg[ , c(value[-1L], NA), by = id]
andmg[ , shift(value, 1, "lead"), id]