4

This code works:

library(plyr)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=FALSE) 

While this code fails:

library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopWorkers(workers)

>Error in do.ply(i) : task 3 failed - "subscript out of bounds"
In addition: Warning messages:
1: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...)’

2: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...)’

I am using R 2.1.12, plyr 1.4 and doSMP 1.0-1. Has anyone figured out a way around this?

edit: In response to Andrie, here is a further illustration:

system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=FALSE)) #1
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=TRUE)) #2
library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=FALSE)) #3
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=TRUE)) #4
stopWorkers(workers)

The first three functions work, but they all take about 3 seconds. Function #2 gives a warning that no parallel backend is registered, and thus executes sequentially. Function #4 gives the same error I referenced in my original post.

/edit: curioser and curiouser: On my mac, the following works:

library(plyr)
library(doMC)
registerDoMC()
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE)

But this fails:

library(plyr)
library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopWorkers(workers)

And this fails too:

library(plyr)
library(snow)
library(doSNOW)
cl <- makeCluster(2, type = "SOCK")
registerDoSNOW(cl)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopCluster(cl)

So I suppose the various parallel back ends for foreach are not interchangeable.

6
  • 1
    The basic problem is that plyr relies on much information being passed in the enclosure on the do.ply function. This data doesn't get passed on by default, and it needs tweaks to the .export parameter to work. Still not sure how to do this in general.
    – hadley
    Commented Apr 7, 2011 at 13:43
  • @hadley: it seems like the doMC package is the only one that works seamlessly with plyr.
    – Zach
    Commented Apr 8, 2011 at 0:01
  • Has anyone succeeded in this?
    – Suraj
    Commented Jul 21, 2011 at 18:17
  • @SFun28 I talked with Hadley (the plyr developer) and he confirmed that plyer only works with the doMP backend. Sorry
    – Zach
    Commented Jul 21, 2011 at 20:16
  • What is doMP, do you mean doMC?
    – user890739
    Commented Apr 7, 2016 at 20:07

3 Answers 3

4

While the question has been answered well by @hadley, I want to add that I think plyr now works with other foreach parallel back-ends. Here is a link to a blog entry containing an example where plyr is used in conjunction with doSNOW:

2
  • From my reading/experimentation, that link suggests that one needs to define a custom createCluster function not provided by snow. not sure why clusterExport doesn't solve this. Can someone confirm that plyr works with each of the parallel backends now without writing a custom function redifining how they export variables?
    – cboettig
    Commented Feb 14, 2012 at 0:37
  • The examples in the link do not work for me. I'm having the same problem.
    – MeloMCR
    Commented Feb 24, 2015 at 18:24
2

Just to confirm @LeeZamparo's answer, plyr does now seem to work with snow, at least on on Windows 7 with R version 2.15.0. The last chunk of code in the question works, though with cryptic warnings:

library(plyr)
library(snow)
library(doSNOW)
cl <- makeCluster(2, type = "SOCK")
registerDoSNOW(cl)

x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)

library(microbenchmark)
mb <- microbenchmark(

      PP <- ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE),
      NP <- ddply(x, .(V), function(df) sum(df$Z),.parallel=FALSE) 
                     )

stopCluster(cl)

Cryptic warnings:

> warnings()
Warning messages:
1: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...

It's not quick, I guess that's the overhead...

> mb
Unit: milliseconds
                                                             expr
1 NP <- ddply(x, .(V), function(df) sum(df$Z), .parallel = FALSE)
2 PP <- ddply(x, .(V), function(df) sum(df$Z), .parallel = TRUE)
        min        lq    median        uq       max
1  11.91518  15.74567  20.10944  23.30453  38.09237
2 314.58008 336.81160 348.42421 358.57337 575.11220

Check it gives the expected result

> PP
  V V1
1 X  4
2 Y  6
3 Z  5

Extra details about this session:

> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252   
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Australia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] microbenchmark_1.1-3 doSNOW_1.0.6         iterators_1.0.6     
[4] foreach_1.4.0        plyr_1.7.1           snow_0.3-10          

loaded via a namespace (and not attached):
[1] codetools_0.2-8 compiler_2.15.0 tools_2.15.0
1

It turns out plyr only works with doMC, but the developer is working on it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.