38

How do I get parallelizaton of code to work in r in Windows? Include a simple example. Posting this self-answered question because this was rather unpleasant to get working. You'll find package parallel does NOT work on its own, but package snow works very well.

6
  • How does the parallel package fail for you? It certainly works for many people, but I think all of the parallel processing packages have more problems on Windows. Commented Jun 6, 2014 at 18:02
  • Doesn't work at all on windows but couldn't tell you why. You have to use snow.
    – Carbon
    Commented Jun 11, 2014 at 7:39
  • @SteveWeston it fails in the sense that it doesn't allow specification of more than one core for processing. stat.ethz.ch/R-manual/R-devel/library/parallel/html/… : "It relies on forking and hence is not available on Windows unless mc.cores = 1."
    – ijoseph
    Commented Jul 18, 2016 at 20:43
  • 2
    @ijoseph That quote only applies to the "mc" functions (such as mclapply, mcMap, mcmapply, and mcparallel). Functions in parallel that were derived from the snow package (such as parLapply, clusterApply, and clusterApplyLB) don't use fork and should execute in parallel on Windows. I've used parLapply from both parallel and snow successfully on Windows, but I've also seen lots of ways that both packages can fail, also. That's why I was curious how exactly it failed. Commented Jul 18, 2016 at 21:05
  • @SteveWeston ah. Didn't realize parallel had those other methods that didn't depend on fork. Will be useful, thanks much!
    – ijoseph
    Commented Jul 18, 2016 at 21:22

6 Answers 6

50

Posting this because this took me bloody forever to figure out. Here's a simple example of parallelization in r that will let you test if things are working right for you and get you on the right path.

library(snow)
z=vector('list',4)
z=1:4
system.time(lapply(z,function(x) Sys.sleep(1)))
cl<-makeCluster(###YOUR NUMBER OF CORES GOES HERE ###,type="SOCK")
system.time(clusterApply(cl, z,function(x) Sys.sleep(1)))
stopCluster(cl)

You should also use library doSNOW to register foreach to the snow cluster, this will cause many packages to parallelize automatically. The command to register is registerDoSNOW(cl) (with cl being the return value from makeCluster()) , the command that undoes registration is registerDoSEQ(). Don't forget to turn off your clusters.

1
  • Thank you, @Carbon! However, this code does not work in R 4.0.4 and later versions. makeCluster just hangs in there without throwing any error. Is there any way to get it worked? Commented May 4, 2022 at 17:51
18

This worked for me, I used package doParallel, required 3 lines of code:

# process in parallel
library(doParallel) 
cl <- makeCluster(detectCores(), type='PSOCK')
registerDoParallel(cl)

# turn parallel processing off and run sequentially again:
registerDoSEQ()

Calculation of a random forest decreased from 180 secs to 120 secs (on a Windows computer with 4 cores).

1
  • what is the difference in using type='PSOCK' instead of type = 'SOCK' ?
    – Nayangar
    Commented Apr 26, 2021 at 8:15
5

Based on the information here I was able to convert the following code into a parallelised version that worked under R Studio on Windows 7.

Original code:

#
# Basic elbow plot function
#
wssplot <- function(data, nc=20, seed=1234){
    wss <- (nrow(data)-1)*sum(apply(data,2,var))
    for (i in 2:nc){
        set.seed(seed)
        wss[i] <- sum(kmeans(data, centers=i, iter.max=30)$withinss)}
    plot(1:nc, wss, type="b", xlab="Number of clusters", 
       ylab="Within groups sum of squares")
}

Parallelised code:

library("parallel")

workerFunc <- function(nc) {
  set.seed(1234)
  return(sum(kmeans(my_data_frame, centers=nc, iter.max=30)$withinss)) }

num_cores <- detectCores()
cl <- makeCluster(num_cores)
clusterExport(cl, varlist=c("my_data_frame")) 
values <- 1:20 # this represents the "nc" variable in the wssplot function
system.time(
  result <- parLapply(cl, values, workerFunc) )  # paralel execution, with time wrapper
stopCluster(cl)
plot(values, unlist(result), type="b", xlab="Number of clusters", ylab="Within groups sum of squares")

Not suggesting it's perfect or even best, just a beginner demonstrating that parallel does seem to work under Windows. Hope it helps.

4

I think these libraries will help you:

foreach (facilitates executing the loop in parallel)
doSNOW (I think you already use it)
doMC (multicore functionality of the parallel package)

May these article also help you

http://vikparuchuri.com/blog/parallel-r-loops-for-windows-and-linux/

http://www.joyofdata.de/blog/parallel-computing-r-windows-using-dosnow-foreach/

1
  • 8
    The doMC package depends on the mclapply function to execute tasks, so it can't execute in parallel on Windows. Commented Jun 6, 2014 at 17:37
3

I'm posting a cross-platform answer here because all the other answers I found were over-complicated for what I needed to accomplish. I'm using an example where I'm reading in all sheets of an excel workbook.

# read in the spreadsheet 
parallel_read <- function(file){

    # detect available cores and use 70%
    numCores  = round(parallel::detectCores() * .70)

    # check if os is windows and use parLapply
    if(.Platform$OS.type == "windows") {

        cl <- makePSOCKcluster(numCores)
        parLapply(cl, file, readxl::read_excel)
        stopCluster(cl)

        return(dfs)

    # if not Windows use mclapply
    } else {

        dfs <-parallel::mclapply(excel_sheets(file), 
                         readxl::read_excel, 
                         path = file,
                         mc.cores=numCores)

        return(dfs)

   }
}
1

For what it is worth. I was running into the same problem but couldn't get any of these to work. I eventually learned that Rstudio has a 'jobs' pane and can run models in the background each on it's own core. so what I did was divy-up my model into 10 segments (it was iterative over 100 vectors so 10 scripts of 10 vectors each) and ran each as a separate job. that way when one finished I could use the output from it immediately and I could keep working on my script without waiting for each model to finish. here is the link all about using jobs https://blog.rstudio.com/2019/03/14/rstudio-1-2-jobs/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.