4

I'd like to add a new column to my data.table, which contains data from one of the other columns. The choice of column, however, varies per row - depending on the contents of another column. So:

for the data set:

     a_data b_data column_choice
[1,]     55      1             a
[2,]     56      2             a
[3,]     57      3             b

generated by:

dat=data.table(a_data = c(55, 56, 57), 
               b_data = c(1,  2,  3), 
               column_choice = c("a", "a", "b"))

I'd like a new column, 'chosen', which contains (per row) either the data from "a_data" or "b_data", depending on the value of "column_choice". The resulting data table will therefore be:

     a_data b_data column_choice chosen
[1,]     55      1             a     55
[2,]     56      2             a     56
[3,]     57      3             b      3

I have managed to get the desired effect using:

dat=dat[, data.table(.SD, chosen=.SD[[paste0(.SD$column_choice, "_data")]]),
        by=1:nrow(a)]
dat$nrow = NULL

however this feels quite clunky; perhaps there's a simpler way to do it (that will no doubt also teach me something about R)?

In practice, the data frame also has lots of other columns that need to be preserved, more choices than just 'a or b', and several of these types of column to generate, so I'd rather not use the basic ifelse solution that may be appropriate for the basic example above.

Thank you very much for your help.

3 Answers 3

4

I think I've now found a properly vectorised one liner, that's also faster than the other answers in this case.

petesFun2 uses data.table aggregation as petesFun, however now vectorised across column_choice (rather than per item, as previously).

While petesFun2 is fine for my purposes, it does leave both the rows and columns in a different order. In the interests of comparison with the other answers, therefore, I've added petesFun2Clean which maintains the same ordering as the other answers.

petesFun2 <-function(myDat) {
  return(myDat[, cbind(.SD, chosen=.SD[[paste0(.BY$column_choice, "_data")]]),
               by=column_choice])
}

petesFun2Clean <-function(myDat) {
  myDat = copy(myDat) # To prevent reference issues
  myDat[, id := seq_len(nrow(myDat))] # Assign an id
  result = myDat[, cbind(.SD, chosen=.SD[[.BY$choice]]),
                 by=list(column_choice, choice=paste0(column_choice, "_data"))]

  # recover ordering and column order.
  return(result[order(id), 
                list(a_data, b_data, c_data, column_choice, chosen)]) 
}

benchmark(benRes<-   myFun(test.dat),
          petesRes<- petesFun(test.dat),
          dowleRes<- dowleFun(test.dat),
          petesRes2<-petesFun2(test.dat),
          petesRes2Clean<- petesFun2Clean(test.dat),
          replications=25,
          columns=c("test", "replications", "elapsed", "relative"))

#                                         test replications elapsed  relative
# 1                  benRes <- myFun(test.dat)           25   0.337  4.160494
# 3             dowleRes <- dowleFun(test.dat)           25   0.191  2.358025
# 5 petesRes2Clean <- petesFun2Clean(test.dat)           25   0.122  1.506173
# 4           petesRes2 <- petesFun2(test.dat)           25   0.081  1.000000
# 2             petesRes <- petesFun(test.dat)           25   4.018 49.604938

identical(petesRes2, benRes)
# FALSE (due to row and column ordering)
identical(petesRes2Clean, benRes)
# TRUE

EDIT: I just noticed (as mentioned by Matthew in the comments) that we now have by group :=. So we can drop the cbind and simply do:

myDat[, chosen := .SD[[paste0(.BY$column_choice, "_data")]], by=column_choice]

2
  • 1
    Doh! Very nice to group by column_choice, +1. There must be a way to avoid the cbind() and reduce the time even further. Great test case for := by group, when implemented.
    – Matt Dowle
    Apr 18, 2012 at 13:06
  • 1
    Nice edit using := by group. Ideally we want to avoid using .SD for efficiency (to save populating .SD with all those columns that aren't needed for each group). Maybe: myDat[, chosen:=myDat[[paste0(column_choice,"_data")]][.I], by=column_choice]. If that works it should be significantly faster as the number of columns of myDat grows.
    – Matt Dowle
    Mar 27, 2013 at 14:10
1

When I think of clunky, things like old bicycles or old cars come to mind, but also doing things in R by iterating over rows. So the below turned out to look clunkier than what you posted in your question, but it goes after a solution in what I think is a more vectorized way. The following appears to be about 10 times faster than (and return identical results as) the sleeker code you posted above.

This suggestion relies on the reshape2 package:

library(data.table)
library(reshape2)

I've added "c" as a possible column_choice to make things a bit more interesting:

dat=data.table(a_data = c(55,56,57,65), 
  b_data = c(1,2,3,4),c_data=c(1000,1001,1002,1003),
  column_choice = c("a", "c", "a", "b"))

Below are the steps, wrapped in a function to prepare them for benchmarking.

myFun<-function(myDat){
# convert data.table to data.frame for melt()ing
  dat1<-data.frame(myDat)
# add ID variable to keep track of things
  dat1$ID<-seq_len(nrow(dat1))
# melt data - because of this line, it's important to only
# pass those variables that are used to select the appropriate value
# i.e., a_data,b_data,c_data,column_choice
  dat2<-melt(dat1,id.vars=c("ID","column_choice"))
# Determine which value to choose: a, b, or c
  dat2$chosen<-as.numeric(dat2$column_choice==substr(dat2$variable,
    1,1))*dat2$value
# cast the data back into the original form
  dat_cast<-dcast(dat2,ID+column_choice~.,
    fun.aggregate=sum,value.var="chosen")
# rename the last variable
  names(dat_cast)[ncol(dat_cast)]<-"chosen"
# merge data back together and return results as a data.table
  datOUT<-merge(dat1,dat_cast,by=c("ID","column_choice"),sort=FALSE)
  return(data.table(datOUT[,c(names(myDat),"chosen")]))
}

Here is your solution packaged into a function:

petesFun<-function(myDat){
  datOUT=myDat[, data.table(.SD,
    chosen=.SD[[paste0(.SD$column_choice, "_data")]]),
    by=1:nrow(myDat)]
  datOUT$nrow = NULL
  return(datOUT)
}

This looks much more elegant than myFun. The benchmarking results show a large difference, however:

Make a larger data.table:

test.df<-data.frame(lapply(dat,rep,100))
test.dat<-data.table(test.df)

and benchmark:

library(rbenchmark)

benchmark(myRes<-myFun(test.dat),petesRes<-petesFun(test.dat),
 replications=25,columns=c("test", "replications", "elapsed", "relative"))
#                             test replications elapsed relative
# 1       myRes <- myFun(test.dat)           25   0.412  1.00000
# 2 petesRes <- petesFun(test.dat)           25   5.429 13.17718

identical(myRes,petesRes)
# [1] TRUE

I propose that "clunky" can be interpreted in different ways :)

6
  • p.s. Can I not melt a data.table? ahhh, 1.8.1 will fix that, apparently.
    – BenBarnes
    Apr 16, 2012 at 21:55
  • Thank you very much for that. I've been looking to understand melting and that was a great help. I'll certainly consider such a method should performance become important.
    – Peter Fine
    Apr 17, 2012 at 15:07
  • However in asking the question, I was wondering if perhaps there was some very simple option (perhaps one elegant line) that will allow a different column to be selected in a vectorised manner. Maybe such a thing doesn't exist?
    – Peter Fine
    Apr 17, 2012 at 15:08
  • @Peter, I'd be interested in an elegant one-liner, too! (I'll keep an eye out here.)
    – BenBarnes
    Apr 17, 2012 at 15:10
  • in case you're interested, I've posted a faster way to do this using just one(ish) data.table operation. Cheers for the melt example though!
    – Peter Fine
    Apr 19, 2012 at 20:52
1

We're starting to use for loops more and more for this kind of task with data.table. Building on Ben's answer and using his benchmark, how about the following :

dowleFun = function(DT) {
  DT = copy(DT)   # Faster to remove this line to add column by reference, but  
                  # included copy() because benchmark repeats test 25 times and
                  # the other tests use the same input table
  w = match(paste0(DT$column_choice,"_data"),names(DT))
  DT[,chosen:=NA_real_]    # allocate new column (or clear it if already exists)
  j = match("chosen",names(DT))     
  for (i in 1:nrow(DT))
      set(DT,i,j,DT[[w[i]]][i])
  DT
}

benchmark(benRes<-myFun(test.dat),
    petesRes<-petesFun(test.dat),
    dowleRes<-dowleFun(test.dat),
    replications=25,columns=c("test", "replications", "elapsed", "relative"),
    order="elapsed")

#                            test replications elapsed relative
# 3 dowleRes <- dowleFun(test.dat)           25    0.30      1.0
# 1      benRes <- myFun(test.dat)           25    0.39      1.3
# 2 petesRes <- petesFun(test.dat)           25    5.79     19.3

If you can remove the copy() then it should be faster and scale better to larger datasets. To test that, perhaps create a very large table and time how long a single run takes.

In this case a simple for loop can be easier to follow.

Having said that, if i could be a 2-column matrix, then A[B] syntax in base could be used (where B contains the row and column positions to select) and it's a one liner :

DT[,chosen:=DT[cbind(1:nrow(DT),paste0(column_choice,"_data"))]]

At the moment you get this :

> DT[cbind(1:3,c(4,4,5))]
Error in `[.data.table`(test.dat, cbind(1:3, c(4, 4, 5))) : 
  i is invalid type (matrix). Perhaps in future a 2 column matrix could return
  a list of elements of DT (in the spirit of A[B] in FAQ 2.14). Please let
  maintainer('data.table') know if you'd like this, or add your comments to
  FR #1611.
4
  • Thank you - very interesting solution, and I didn't realise that data.table could pass by reference (although I don't fully understand how the references work yet, it seems to depend on whether you've used any transformations other than := or not, in my initial tests?).
    – Peter Fine
    Apr 18, 2012 at 12:13
  • @Peter Not sure what you're asking here. Do ?":=", ?copy and their example sections help? Also search the data.table tag for "reference" or ":=".
    – Matt Dowle
    Apr 18, 2012 at 12:58
  • I realise now that, after newDT = DT, newDT[2, col1:= 5] will also affect DT, since the operation is by reference. But, if in between those two operations I do something else, such as newDT$col2[2]=5, then afterwards, changes to newDT (even those using :=) are no longer reflected in DT. So does newDT$col2[2]=5 somehow break the reference? Perhaps this should have been a seperate question...
    – Peter Fine
    Apr 18, 2012 at 14:11
  • @Peter Good question. Yes, new question please.
    – Matt Dowle
    Apr 18, 2012 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.