-5

I'm newbie with Spark. I'm trying to read the code and to understand how K-means in Spark Streaming works. I do not know how can to get the number of iterations that the algorithm performs in the same data's group. I can't find the Java file with this information.

Can you help me, please?

Thank you

Solution: In this file /spark-1.5.0/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala there is a while statement in the run method that use a variable called iteration and Spark writes it in a log for each run.

0

2 Answers 2

2

Just as a small addition to majitux' solution (I am not allowed to comment yet). If you want to know the number of iterations K-Means takes simply change the log level of SPARK to INFO. Either inside the shell using:

spark.sparkContext.setLogLevel("INFO")

Or by setting it as default inside the conf/log4j.properties.

After K-Means finished running the string "KMeans++ converged in X iterations" will appear in the log.

1

When you initialize KMeans class, you can specify max-iteration parameters.

new KMeans().setMaxIterations(iterations)

then it will use that parameter for each prediction

2
  • Yes, I know. But sometimes algorithm performs fewer iterations than the maximum, so I need to know that number.
    – mjbsgll
    Oct 2, 2015 at 19:57
  • I found the answer!: In this file /spark-1.5.0/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala there is a while in the run method that use a variable called iteration and Spark writes it in a log for each run.
    – mjbsgll
    Oct 20, 2015 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.