I created an array using SparkContext
using 2 partitions, I also try to use mapPartition
to handle the elements, however I've got a very strange error, when I code like this:
val masterURL = "local[*]"
val conf = new SparkConf().setAppName("KMeans Test").setMaster(masterURL)
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
val data = sc.textFile("file:/d:/data/kmeans_data.txt")
val parsedData = data.mapPartitions(partition => parseData(partition)).cache()
parsedData.mapPartitions(points =>
points.map(point =>
println(point)
)
)
It has no errors, however, when I replace map to foreach, it tips an error:
parsedData.mapPartitions(points =>
points.foreach(point =>
println(point)
)
)
the error is as follows:
Type mismatch, expected: (Iterator[Vector]) => Iterator[NotInferedU], actual: (Iterator[Vector]) => Unit Expression of type Unit doesn't conform to expected type Iterator[U_]
In addition, the first code snip also do not print anything in console panel, why?
Unit
. Try returning thepoints
Iterator inmapPartitions
. – Álvaro Valencia Jul 30 '18 at 7:06