I am new to Scala. How come the "map" function is not serializable? How to make it serializable? For example, if my code is like below:
val data = sc.parallelize(List(1,4,3,5,2,3,5))
def myfunc(iter: Iterator[Int]) : Iterator[Int] = {
val lst = List(("a", 1),("b", 2),("c",3), ("a",2))
var res = List[Int]()
while (iter.hasNext) {
val cur = iter.next
val a = lst.groupBy(x => x._1).mapValues(_.size)
//val b= a.map(x => x._2)
res = res ::: List(cur)
}
res.iterator
}
data.mapPartitions(myfunc).collect
If I uncomment the line
val b= a.map(x => x._2)
The code returns an exception:
org.apache.spark.SparkException: Task not serializable
Caused by: java.io.NotSerializableException: scala.collection.immutable.MapLike$$anon$2
Serialization stack:
- object not serializable (class: scala.collection.immutable.MapLike$$anon$2, value: Map(1 -> 3))
- field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: a, type: interface scala.collection.immutable.Map)
Thank you very much.
lst
really just a plain list in the actual code? Or another RDD?a
orb
from the function.