0

I'm reading csv file in directory with more than 100 files, then I'm doing some stuff, I have 8 cores cpu so I want to do in parallel mode to finish faster.

I wrote some code but it doesn't work for me - (using linux)

library(data.table)
library(parallel)

# Calculate the number of cores
no_cores <- detectCores() - 1
# Initiate cluster
cl <- makeCluster(no_cores)

processFile <- function(f) {

  # reading file by data.table 
  df <- fread(f,colClasses = c(NA,NA, NA,"NULL", "NULL", "NULL"))

  A <- parLapply(cl,sapply(windows, function(w) {return(numOverlaps(w,df))}))

  stopCluster(cl)
}

files <- dir("/home/shared/", recursive=TRUE, full.names=TRUE, pattern=".*\\.txt$")

# Apply the function to all files.

 result <- sapply(files, processFile)

As you see I want to run function in processFile(A) but it doesn't work!

How it's possible to run that function in parallel processing mode?

1 Answer 1

2

You have the concept on its head. You need to pass parLapply the list of files and then work on them. The anonymous function should do the entire process of processing the individual file and returning the desired result.

My suggestion would be to first make this work using regular lapply or sapply and only then power up parallel backend, export all necessary libraries and objects you may need.

parLapply(cl, X = files, FUN = function(x, ...) {
  ... code for processing the file
})
4
  • It doesn't work for me because I'm using "data.table" package and when reading table by "fread" (inside parLapply function ) I got error that I can't read file!
    – Big Data
    Commented Mar 13, 2017 at 12:39
  • @BigData It's hard to say without (reproducible) code. If you provide a chunk of code (generate data, write to disk, read with fread...) I can perhaps try any make it work in parallel. Commented Mar 14, 2017 at 8:47
  • I updated code which I'm reading by fread function, I mean I included this line inside your function - the error I have is: Error in checkForRemoteErrors(val) : 7 nodes produced errors; first error: could not find function "fread" -
    – Big Data
    Commented Mar 14, 2017 at 9:33
  • 2
    Think about the background R workers as fresh R sessions (started as R --vanilla). In a fresh R session, how do you make sure fread() is found? ... by using library("data.table"), correct? You can either add that to your function, or use clusterEvalQ() before parLapply(). Alternatively, you can call the function using data.frame::fread().
    – HenrikB
    Commented Mar 17, 2017 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.