I want to create a new mongodb RDD each time I enter inside foreachRDD
. However I have serialization issues:
mydstream
.foreachRDD(rdd => {
val mongoClient = MongoClient("localhost", 27017)
val db = mongoClient(mongoDatabase)
val coll = db(mongoCollection)
// ssc is my StreamingContext
val modelsRDDRaw = ssc.sparkContext.parallelize(coll.find().toList) })
This will give me an error:
object not serializable (class: org.apache.spark.streaming.StreamingContext, value: org.apache.spark.streaming.StreamingContext@31133b6e)
Any idea?
SparkContext
is not serializable, so you cannot use inside any transformation or action methods, you have to use only in driver class.