0

I am using split() for split a dataset of more than 15M lines. The common split() works but consumes a lot of time. Then I made a function to split the dataset by number of rows for, then, give that resulted list to be splitted (this is the essential split, a split by one of the variables) in parallel using snow.

It works a lot faster now but for 15M lines it crashs because this model consumes a lot of memory (I am using 3 machines of 16GB memory each).

Do you know any alternative for doing this split without using so much memory? I tried ff package but the split function doesnt work equally as if worked with data.frame. I tried the split-apply-combine of ffbase package but I just need the split part and I didnt find a way to use only the split step.

Thank you

4
  • Seems like your splitting activity, will almost certainly create a copy of the data. That is probably too much to hold in memory. I'd probably try to process small chunks one at a time from the data, instead of create all small chunks at once, then processing them.
    – cory
    Mar 5, 2015 at 12:59
  • I am trying with 3 machines using one core each. Do you mean to stop parallel processing? Then it will take a lot of time again. Mar 5, 2015 at 19:18
  • ffdfdply from package ffbase should return a data.frame inside the FUN argument. If this is not the case in your example, you could try to mimic the internals of ffdfdply which basically sets up a list of positions, one for each split group and saves this in ff. If your split positions are possible in RAM, why don't you just put this 1 column in RAM and use which to select your data from ff into RAM. Hope this helps.
    – user1600826
    Mar 9, 2015 at 9:09
  • Thank you for helping @jwijffels. I tried which but it is very slow. Can you explain with more details what you suggested? I also built a list of ffdf (transforming a list of data.frame in a list of ffdf) and tried to apply this in parApply() of snow but I get the error: nodes produced errors; first error: incorrect number of dimensions. The structure of ffdf is different from data.frame. Mar 11, 2015 at 18:01

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.