0

When launching a spark cluster via sparklyr, I notice that it can take between 10-60 seconds for all the executors to come online.

Right now I'm using Sys.sleep(60) to allow time for them to come online, but sometimes it takes longer than that, and sometimes it is shorter than that. I want a programmatic way to adjust for this time variance, similar to this question regarding Python. So I think I want to pass getExecutorMemoryStatus via sparklyr, but I'm not sure how to do this.

To see what I'm seeing, run the following code to launch a yarn-client spark connection, and check the Yarn UI. In the Event Timeline we can see at which time each executor comes online.

spark_config <- spark_config()
spark_config$spark.executor.memory <- "11G"
spark_config$`sparklyr.shell.driver-memory` <- "11G"
spark_config$spark.dynamicAllocation.enabled <- FALSE
spark_config$`spark.yarn.executor.memoryOverhead` <- "1G"
spark_config$spark.executor.instances <- 32

sc <- spark_connect(master = "yarn-client", config = spark_config)

1 Answer 1

2

So I think I want to pass getExecutorMemoryStatus via sparklyr, but I'm not sure how to do this.

You have to retrieve SparkContext object:

sc <- spark_connect(...)

spark_context(sc) %>%
  ...

and then invoke the method:

 ... %>% invoke("getExecutorMemoryStatus")

Together:

spark_context(sc) %>% 
  invoke("getExecutorMemoryStatus") %>% 
  names()

should give you a list of active executors.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.