I have written some code using foreach
which processes and combines a large number of CSV files. I am running it on a 32 core machine, using %dopar%
and registering 32 cores with doMC
. I have set .inorder=FALSE
, .multicombine=TRUE
, verbose=TRUE
, and have a custom combine function.
I notice if I run this on a sufficiently large set of files, it appears that R attempts to process EVERY file before calling .combine the first time. My evidence is that in monitoring my server with htop, I initially see all cores maxed out, and then for the remainder of the job only one or two cores are used while it does the combines in batches of ~100 (.maxcombine
's default), as seen in the verbose console output. What's really telling is the more jobs i give to foreach, the longer it takes to see "First call to combine"!
This seems counter-intuitive to me; I naively expected foreach to process .maxcombine
files, combine them, then move on to the next batch, combining those with the output of the last call to .combine
. I suppose for most uses of .combine
it wouldn't matter as the output would be roughly the same size as the sum of the sizes of inputs to it; however my combine function pares down the size a bit. My job is large enough that I could not possibly hold all 4200+ individual foreach job outputs in RAM simultaneously, so I was counting on my space-saving .combine
and separate batching to see me through.
Am I right that .combine doesn't get called until ALL my foreach
jobs are individually complete? If so, why is that, and how can I optimize for that (other than making the output of each job smaller) or change that behavior?
foreach
will distribute lists in bundles of length(parList)/ncores. I suppose combining partial results might be useful as a space saving measure. You may get more control with theiter
function.