I am a beginner at Apache Spark and I'm running it in standalone mode. The task dataframe.count()
is hanging.
SparkConf conf = new SparkConf();
conf.set("spark.driver.allowMultipleContexts", "true");
conf.set("spark.executor.memory", "10g");
conf.set("spark.dirver.maxResultSize","10g");
conf.set("spark.driver.memory ", "10g");
//Initialize sparkcontext
Dataframt dt = //load data from reshift
JavaRDD<String> rdd = sc.textFile(url);
JavaPairRDD<String, String> pairRdd = rdd.mapToPair(SparkFunctionsImpl
.strToMap());
//dt.count()
//pairrdd => map => collectAsMap()
Spark task hangs at count()
and collectAsMap()
and doesn't proceed from there.
Looks like rdd.collectAsMap
and dataframe.count()
are executing in parallel and spark is hanging with none of the tasks proceeding.