4

I could do this by looping through my dataset multiple times but thought there must be a more efficient way to do this through data.table. This is what the dataset looks like:

CaseID         Won     OwnerID      Time_period    Finished
  1            yes        A              1              no
  1            yes        A              3              no
  1            yes        A              5              yes
  2            no         A              4              no
  2            no         A              6              yes
  3            yes        A              2              yes
  4            no         A              3              yes
  5            15         B              2              no

For each row, by owner, I want to generate an average of the amount of cases finished before that time period that are won.

CaseID         Won     OwnerID      Time_period     Finished     AvgWonByOwner  
  1            yes        A              1              no            NA
  1            yes        A              3              no             1
  1            yes        A              5              yes           .5
  2            no         A              4              no            .5
  2            no         A              6              yes           2/3
  3            yes        A              2              yes           NA
  4            no         A              3              yes           1
  5            15         B              2              no            NA

Looking at this in detail, it seems ridiculously complicated. I thought you might be able to do this with some sort of rolling merge, but I don't know how to set a condition where the average is only calculated from Won before the date of the row and where it has to have the same ownerID.

Edit 1: Explanation for numbers in final column

AvgWonByOwner          Explanation
   NA                  t = 1, No cases finished yet, this could be 0 too
   1                   t = 3, case 3 finished and is won, so average wins is 1
  .5                   t = 5, case 3 finished, won; case 4 finished lost; average = .5
  .5                   t = 4, case 3 finished, won; case 4 finished lost; average = .5
  2/3                  t = 6, case 3 finished, won, case 4 finished lost, case 1 finished won, average: 2/3
   NA                  t = 1, No cases finished yet, this could be 0 too
   1                   t = 3, case 3 finished and is won, so average wins is 1
   NA                  t = 1, No cases finished yet, this could be 0 too
3
  • 1
    you need to explain how you arrived at those numbers in the last column
    – eddi
    Nov 12, 2013 at 21:44
  • Case 1 is finished at point t=5, it doesn't get counted in average until t = 6. Basically it's a strict inequality.
    – Luke
    Nov 12, 2013 at 22:27
  • @Luke +1 for a real head scratcher! Nov 12, 2013 at 22:29

2 Answers 2

4
dt = data.table(structure(list(CaseID = c(1, 1, 1, 2, 2, 3, 4, 5), Won = structure(c(3L, 
3L, 3L, 2L, 2L, 3L, 2L, 1L), .Label = c("15", "no", "yes"), class = "factor"), 
    OwnerID = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L), .Label = c("A", 
    "B"), class = "factor"), Time_period = c(1L, 3L, 5L, 4L, 
    6L, 2L, 3L, 2L), Finished = structure(c(1L, 1L, 2L, 1L, 2L, 
    2L, 2L, 1L), .Label = c("no", "yes"), class = "factor")), .Names = c("CaseID", 
"Won", "OwnerID", "Time_period", "Finished"), row.names = c(NA, 
-8L), class = c("data.table", "data.frame")))

# order
setkey(dt, OwnerID, Time_period)

# calculate the required ratio but including current time
dt[, ratio := cumsum(Finished == "yes" & Won == "yes") /
              cumsum(Finished == "yes"),
     by = list(OwnerID)]

# shift to satisfy the strict inequality as per OP
dt[, avgWon := c(NaN, ratio[-.N]), by = OwnerID]

# take the first one for each time (that is last one from previous time)
# so that all of the outcomes happening at same time are accounted for
dt[, avgWon := avgWon[1], by = key(dt)]

dt[order(OwnerID, CaseID)]
#   CaseID Won OwnerID Time_period Finished     ratio    avgWon
#1:      1 yes       A           1       no       NaN       NaN
#2:      1 yes       A           3       no 1.0000000 1.0000000
#3:      1 yes       A           5      yes 0.6666667 0.5000000
#4:      2  no       A           4       no 0.5000000 0.5000000
#5:      2  no       A           6      yes 0.5000000 0.6666667
#6:      3 yes       A           2      yes 1.0000000       NaN
#7:      4  no       A           3      yes 0.5000000 1.0000000
#8:      5  15       B           2       no       NaN       NaN
10
  • +1 I have no idea how you managed to crack this. Very clever. Nov 13, 2013 at 0:12
  • I'm not sure this will work if, for instance, there are multiple records with the same Time_period for a single OwnerID. (The problem is in that "shift" step, which takes a bit of a shortcut...) Nov 13, 2013 at 0:18
  • @JoshO'Brien I think it will - there is a step in there to address specifically that issue, but if you think I made a mistake and have an example where it doesn't work I'd love to see it
    – eddi
    Nov 13, 2013 at 0:29
  • @eddi -- Well, I'm doubting myself because it was just my intuition and because it is you. But I'll take another minute or two to see where I went wrong ;) Nov 13, 2013 at 0:32
  • 1
    @eddi -- Mine just errors out because dt's key gets dropped when doing dt <- rbind(dt, dt[3,]). Reset the key and it works for this (admittedly unimportant) case. I see now, though, what dt[, avgWon := avgWon[1], by = key(dt)] does, and why yours thus works. Very clever (even if it is a bit opaque)! Nov 13, 2013 at 3:55
2
## Compute a data.table recording the win percentage at end of each time period
B <- dt[Finished=="yes",]
B[,winpct := (cumsum(Won=="yes")/seq_along(Won)),by=OwnerID]

## Shift forward by one time step, as per OP's description of problem
B[,Time_period := Time_period + 1]
setkeyv(B, key(dt))

## Append win percentage column back to original data.table
cbind(dt, AvgWonByOwner=B[dt, winpct, roll=TRUE][["winpct"]])
#    CaseID Won OwnerID Time_period Finished AvgWonByOwner
# 1:      1 yes       A           1       no            NA
# 2:      3 yes       A           2      yes            NA
# 3:      1 yes       A           3       no     1.0000000
# 4:      4  no       A           3      yes     1.0000000
# 5:      2  no       A           4       no     0.5000000
# 6:      1 yes       A           5      yes     0.5000000
# 7:      2  no       A           6      yes     0.6666667
# 8:      5  15       B           2       no            NA

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.