11

I have a data set with several time assessments for each participant. I want to select the last assessment for each participant. My dataset looks like this:

ID  week  outcome
1   2   14
1   4   28
1   6   42
4   2   14
4   6   46
4   9   64
4   9   71
4  12   85
9   2   14
9   4   28
9   6   51
9   9   66
9  12   84

I want to select only the last observation/assessment for each participant, but I only have the number of weeks as an indicator for each participant. How is this possible to do in R (or excel?)

thanks in advance,

niki

1
  • 1
    As an aside, please make sure you're doing something sensible with this data; just taking the last available assessment can lead to very wrong inferences, depending on why you're missing data and what you're looking for. Mar 1, 2012 at 12:06

6 Answers 6

11

Here is one base-R approach:

do.call("rbind", 
        by(df, INDICES=df$ID, FUN=function(DF) DF[which.max(DF$week), ]))
  ID week outcome
1  1    6      42
4  4   12      85
9  9   12      84

Alternatively, the data.table package offers a succinct and expressive language for performing data frame manipulations of this type:

library(data.table)
dt <- data.table(df, key="ID")

dt[, .SD[which.max(outcome), ], by=ID] 
#      ID week outcome
# [1,]  1    6      42
# [2,]  4   12      85
# [3,]  9   12      84

# Same but much faster. 
# (Actually, only the same as long as there are no ties for max(outcome)..)
dt[ dt[,outcome==max(outcome),by=ID][[2]] ]   # same, but much faster.

# If there are ties for max(outcome), the following will still produce
# the same results as the method using .SD, but will be faster
i1 <- dt[,which.max(outcome), by=ID][[2]]
i2 <- dt[,.N, by=ID][[2]]
dt[i1 + cumsum(i2) - i2,]

Finally, here is a plyr-based solution

library(plyr)

ddply(df, .(ID), function(X) X[which.max(X$week), ])
#   ID week outcome
# 1  1    6      42
# 2  4   12      85
# 3  9   12      84
10
  • Nice responses. I was trying to think of how to do this with plyr or aggregate and fail because I can't figure out how to return outcome with out hacking it together. +1 Feb 27, 2012 at 21:01
  • @TylerRinker -- Would you take a look at the plyr solution I just added? I mostly end up using base-R or data.table, so might be missing some obvious improvement to it. Thanks! Feb 27, 2012 at 21:05
  • @JoshO'Brien Thank you for your responses. Works great. Feb 27, 2012 at 23:19
  • @MatthewDoyle -- Very clever. As I familiarize myself with data.table, I'm noticing that many of the speediest constructs use a couple of dt[] operations instead of just one. In other applications, I'd seen this with dt[dt[], ] and dt[]dt[], but this (using the nested call to get a logical vector) is new. (Think I may have seen something similar using dt[dt[,,which=TRUE]]). Anyways, thanks for the suggestion and (as I've said before) do feel free to edit any of my data.table related posts at any time. Cheers. Mar 1, 2012 at 13:39
  • @Josh Hi. I remember your previous comments about editing so I felt confident to go ahead this time, great. Btw my surname is with w not y (I don't mind, just saying because the @ didn't work because of that, I just saw your comment by chance). This particular trick isn't the prettiest, is it. Have some ideas to improve: .I would be nice to add, and/or being a bit cleverer with .SD so that the natural syntax you used is just fast anyway as it should be, ideally. That's one reason a lot is packed into [] so that internally we can do better optimisation in the future.
    – Matt Dowle
    Mar 1, 2012 at 17:01
10

If you're just looking for the last observation per person ID, then a simple two line code should do it. I am up always for simple base solution when possible while it is always great to have more than one ways to solve a problem.

dat[order(dat$ID,dat$Week),]  # Sort by ID and week
dat[!duplicated(dat$ID, fromLast=T),] # Keep last observation per ID

   ID Week Outcome
3   1    6      42
8   4   12      85
13  9   12      84
1
  • This is a high quality answer that deserves more up-votes. When it comes to R, I appreciate answers that don't involve installing new packages, etc. Thanks for adding your two cents.
    – ChrisP
    Sep 1, 2013 at 20:29
2

I can play this game. I ran some benchmarks on differences between lapply, sapply, and by, among other things. It appears to me that the more you're in control of data types and the more basic the operation, the faster it is (e.g., lapply is generally faster than sapply, and as.numeric(lapply(...)) is going to be faster, also). With that in mind, this produced the same results as above and may be faster than the rest.

df[cumsum(as.numeric(lapply(split(df$week, df$id), which.max))), ]

Explanation: we only want which.max on the week per each id. That handles the contents of lapply. We only need the vector of these relative points, so make it numeric. The result is the vector (3, 5, 5). We need to add the positions of the prior maxes. This is accomplished with cumsum.

It should be noted, this solution is not general when I use cumsum. It may require that prior to execution we sort the frame on id and week. I hope you understand why (and know how to use with(df, order(id, week)) in the row index to achieve that). In any case, it may still fail if we don't have a unique max, because which.max only takes the first one. Therefore, my solution is a bit question begging, but that goes without saying. We're trying to extract very specific information for a very specific example. Our solutions can't be general (even though the methods are important to understand generally).

I'll leave it to trinker to update his comparisons!

2

Another option in base: df[rev(rownames(df)),][match(unique(df$ID), rev(df$ID)), ]

6
  • great thanks, first one works perfect. using the second i get some cases replicated, no clue why. Feb 27, 2012 at 23:40
  • Just here for interest... I'd stick with Josh's! Will look at the dupes issue though..
    – jbaums
    Feb 27, 2012 at 23:51
  • grep wasn't really appropriate for exact matching of numerics.. x==df$ID does a better job.
    – jbaums
    Feb 28, 2012 at 0:12
  • Was this intended to get the (ID, week) combos for (1, 6), (4, 12), and (9, 12) like Josh (and myself) obtained in our versions? I got the lowest (first) week values from your second command: (1, 2), (4, 2), and (9, 2). Feb 29, 2012 at 23:35
  • jbaums the first answer works but the second response does not return the correct response. Mar 1, 2012 at 0:20
2

This answer uses the data.table package. It should be very fast, even with larger data sets.

setkey(DT, ID, week)              # Ensure it's sorted.
DT[DT[, .I[.N], by = ID][, V1]]

Explanation: .I is an integer vector holding the row locations for the group (in this case the group is ID). .N is a length-one integer vector containing the number of rows in the group. So what we're doing here is to extract the location of the last row for each group, using the "inner" DT[.], using the fact that the data is sorted according to ID and week. Afterwards we use that to subset the "outer" DT[.].

For comparison (because it's not posted elsewhere), here's how you can generate the original data so that you can run the code:

DT <- 
  data.table(
    ID = c(rep(1, 3), rep(4, 5), rep(9, 5)),
    week = c(2,4,6, 2,6,9,9,12, 2,4,6,9,12), 
    outcome = c(14,28,42, 14,46,64,71,85, 14,28,51,66,84))
1

I've been trying to use split and tapply a bit more to become more acquainted with them. I know this question have been answered already but I thought I'd add another solotuion using split (pardon the ugliness; I'm more than open to feedback for improvement; thought maybe there was a use to tapply to lessen the code):

sdf <-with(df, split(df, ID))
max.week <- sapply(seq_along(sdf), function(x) which.max(sdf[[x]][, 'week']))
data.frame(t(mapply(function(x, y) y[x, ], max.week, sdf)))

I also figured why we have 7 answers here it was ripe for a benchmark. The results may surprise you (using rbenchmark with R2.14.1 on a Win 7 machine):

# library(rbenchmark)
# benchmark(
#     DATA.TABLE= {dt <- data.table(df, key="ID")
#         dt[, .SD[which.max(outcome),], by=ID]},
#     DO.CALL={do.call("rbind", 
#         by(df, INDICES=df$ID, FUN=function(DF) DF[which.max(DF$week),]))},
#     PLYR=ddply(df, .(ID), function(X) X[which.max(X$week), ]),
#     SPLIT={sdf <-with(df, split(df, ID))
#         max.week <- sapply(seq_along(sdf), function(x) which.max(sdf[[x]][, 'week']))
#         data.frame(t(mapply(function(x, y) y[x, ], max.week, sdf)))},
#     MATCH.INDEX=df[rev(rownames(df)),][match(unique(df$ID), rev(df$ID)), ],
#     AGGREGATE=df[cumsum(aggregate(week ~ ID, df, which.max)$week), ],
#     #WHICH.MAX.INDEX=df[sapply(unique(df$ID), function(x) which.max(x==df$ID)), ],
#     BRYANS.INDEX = df[cumsum(as.numeric(lapply(split(df$week, df$ID), 
#         which.max))), ],
#     SPLIT2={sdf <-with(df, split(df, ID))
#         df[cumsum(sapply(seq_along(sdf), function(x) which.max(sdf[[x]][, 'week']))),
#         ]},
#     TAPPLY=df[tapply(seq_along(df$ID), df$ID, function(x){tail(x,1)}),],
# columns = c( "test", "replications", "elapsed", "relative", "user.self","sys.self"), 
# order = "test", replications = 1000, environment = parent.frame())

          test replications elapsed  relative user.self sys.self
6    AGGREGATE         1000    4.49  7.610169      2.84     0.05
7 BRYANS.INDEX         1000    0.59  1.000000      0.20     0.00
1   DATA.TABLE         1000   20.28 34.372881     11.98     0.00
2      DO.CALL         1000    4.67  7.915254      2.95     0.03
5  MATCH.INDEX         1000    1.07  1.813559      0.51     0.00
3         PLYR         1000   10.61 17.983051      5.07     0.00
4        SPLIT         1000    3.12  5.288136      1.81     0.00
8       SPLIT2         1000    1.56  2.644068      1.28     0.00
9       TAPPLY         1000    1.08  1.830508      0.88     0.00

Edit1: I omitted the WHICH MAX solution as it does not return the correct results and returned an AGGREGATE solution as well that I wanted to use (compliments of Bryan Goodrich) and an updated version of split, SPLIT2, using cumsum (I liked that move).

Edit 2: Dason also chimed in with a tapply solution I threw into the test that fared pretty well too.

2
  • Although to make the tapply solution work for what the OP wanted you'd technically have to sort by week if it wasn't already sorted. In this case it was sorted by week so oh well.
    – Dason
    Mar 1, 2012 at 0:34
  • 1
    Huh? This table appears repeat each test 1000 times on a very small dataset. Why are the results of any use (in any way whatsoever) in practice? You need to run a single test of each method, on a large dataset. That's what matters.
    – Matt Dowle
    Mar 1, 2012 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.