I am trying to find a fit hardware size for my Spark job. My understanding was that scaling up the number of machines may help speeding up my job, considering the fact that my job does not have a complex action operation and therefore probably small amount of calculation in the driver program. However, what I observe is that the job execution speed lowers down when adding resources to Spark. I can reproduce this effect using the following simple job:
- Loading a text file (~100Gb) from HDFS
Running a simple 'filter' transformation on the RDD, that looks like below:
JavaRDD<String> filteredRDD = rdd.filter(new Function<String, Boolean>() { public Boolean call(String s) { String filter = "FILTER_STRING"; return s.indexOf(filter) > 0 ? true : false; }
- Running count() action on the result
The scaling problem shows itself when I scale up the number of machines in the cluster from 4 to 8. Here are some details about the environment:
- Each executor is configured to use 6 GB of memory. Also the HDFS is co-hosted on the same machines.
- Each machine has 24 GB of RAM in total and 12 cores (configured to use 8 for Spark executors).
- Spark is hosted in a YARN cluster.
Any ideas why I am not getting the degree of scalabilty I expect from Spark?