1

I have a large data.table (circa 900k rows) which can be represented by the following example:

        row.id entity.id event.date result
 1:      1       100 2015-01-20     NA
 2:      2       101 2015-01-20     NA
 3:      3       104 2015-01-20     NA
 4:      4       107 2015-01-20     NA
 5:      5       103 2015-01-23     NA
 6:      6       109 2015-01-23     NA
 7:      7       102 2015-01-23     NA
 8:      8       101 2015-01-26     NA
 9:      9       110 2015-01-26     NA
10:     10       112 2015-01-26     NA
11:     11       109 2015-01-26     NA
12:     12       130 2015-01-29     NA
13:     13       100 2015-01-29     NA
14:     14       127 2015-01-29     NA
15:     15       101 2015-01-29     NA
16:     16       119 2015-01-29     NA
17:     17       104 2015-02-03     NA
18:     18       101 2015-02-03     NA
19:     19       125 2015-02-03     NA
20:     20       130 2015-02-03     NA

Essentially I have columns containing: the ID representing the entity in question (entity.id); the date of an event in which this ID partook (note that many, and differing numbers of, entities will participate in each event). I need to calculate a factor that, for each entity.id on each event date, depends (non-linearly) on the time (in days) that has elapsed since all the previous events in which that entity ID was entered.

To put it in other, more programmatic terms, on each row of the data.table I need to find all instances with matching ID and where the date is older than the event date of the row in question, work out the difference in time (in days) between the ‘current’ and historical events, and sum some non-linear function applied to each of the time periods (I’ll use the square in this example).

In the example above, for entity.id = 101 on 03-02-2015 (row 18), the we would need to look back to that ID's prior entries on rows 15, 8 and 2, calculate the differences in days from the ‘current’ event (14, 8 and 5 days), and then calculate the answer by summing the squares of those periods (14^2 + 8^2 + 5^2) = 196 + 64 + 25 = 285. (The real function is somewhat more complex but this is sufficiently representative.)

This is trivial to achieve with for-loops, as per below:

# Create sample dt
dt <- data.table(row.id = 1:20,
     entity.id = c(100, 101, 104, 107, 103, 109, 102, 101, 110, 112,
                   109, 130, 100, 127, 101, 119, 104, 101, 125, 130),
     event.date = as.Date(c("2015-01-20", "2015-01-20", "2015-01-20", "2015-01-20", 
                    "2015-01-23", "2015-01-23", "2015-01-23",
                    "2015-01-26", "2015-01-26", "2015-01-26", "2015-01-26",
                    "2015-01-29", "2015-01-29", "2015-01-29", "2015-01-29", "2015-01-29",
                    "2015-02-03", "2015-02-03", "2015-02-03", "2015-02-03")),
     result = NA)
setkey(dt, row.id)

for (i in 1:nrow(dt)) { #loop through each entry

  # get a subset of dt comprised of rows with this row's entity.id, which occur prior to this row
  event.history <- dt[row.id < i & entity.id == entity.id[i]]

  # calc the sum of the differences between the current row event date and the prior events dates, contained within event.history, squared
  dt$result[i] <- sum( (as.numeric(dt$event.date[i]) - as.numeric(event.history$event.date)) ^2 )
}

Unfortunately, on the real dataset it is also extremely slow, no doubt because if the amount of subsetting operations required. Is there a way to vectorise, or otherwise speed up, this operation? I’ve searched and searched and wracked my brains but can’t work out how to vecotrally subset rows based on differing data per each row without looping.

Note that I created a row.id column to allow me to extract all prior rows (rather than prior dates), as the two are broadly equivalent (an entity cannot attend more than one event a day) and this way was much quicker (I think because it avoids the need to coerce the dates to numeric before doing the comparison, ie. Dt[as.numeric(event_date) < as.numeric(event_date[i])]

Note also that I’m not wedded to it being a data.table; I’m happy to use dplyr or other mechanisms to achieve this if need be.

1
  • Thank you all for the rapid responses, I never expected to get suggestions - and an elegant solution - so quickly!
    – WaveyDavey
    Sep 19, 2019 at 6:23

1 Answer 1

1

I think this can be achieved using a self-join with appropriate non-equi join critieria:

dt[, result2 := dt[
                   dt,
                   on=c("entity.id","event.date<event.date"),
                   sum(as.numeric(x.event.date - i.event.date)^2), by=.EACHI]$V1
                  ]
dt

This gives a result which matches your output from the loop, with the exception of the NA values:

#    row.id entity.id event.date result result2
# 1:      1       100 2015-01-20      0      NA
# 2:      2       101 2015-01-20      0      NA
# 3:      3       104 2015-01-20      0      NA
# 4:      4       107 2015-01-20      0      NA
# 5:      5       103 2015-01-23      0      NA
# 6:      6       109 2015-01-23      0      NA
# 7:      7       102 2015-01-23      0      NA
# 8:      8       101 2015-01-26     36      36
# 9:      9       110 2015-01-26      0      NA
#10:     10       112 2015-01-26      0      NA
#11:     11       109 2015-01-26      9       9
#12:     12       130 2015-01-29      0      NA
#13:     13       100 2015-01-29     81      81
#14:     14       127 2015-01-29      0      NA
#15:     15       101 2015-01-29     90      90
#16:     16       119 2015-01-29      0      NA
#17:     17       104 2015-02-03    196     196
#18:     18       101 2015-02-03    285     285
#19:     19       125 2015-02-03      0      NA
#20:     20       130 2015-02-03     25      25
3
  • Couldn't you do the join directly? I'm surprised with the join in the j statement.
    – Cole
    Sep 19, 2019 at 0:28
  • Feel free to roll-back. Updating by reference didn't work with the alternative approach. It updated all non-equi matches to the final sum. So row 1 was 81 and row 2 was 285 and row 3 was 196.
    – Cole
    Sep 19, 2019 at 2:16
  • Thank you @thelatemail, this looks to be perfect!
    – WaveyDavey
    Sep 19, 2019 at 6:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.