0

I am trying to use ForeachWriter interface in Spark 2.1 it's interface, but I cannot use it.

2 Answers 2

1

It will be supported in Spark 2.2.0. To learn how to use it, I suggest you read this blog post: https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html

You can try Spark 2.2.0 RC2 [1] or just wait for the final release.

Another option is taking a look at this blog if you cannot use Spark 2.2.0+:

https://databricks.com/blog/2017/04/04/real-time-end-to-end-integration-with-apache-kafka-in-apache-sparks-structured-streaming.html

It has a very simple Kafka sink and maybe that's enough for you.

[1] http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Apache-Spark-2-2-0-RC2-td21497.html

0

First thing to know is that, if you working with spark structured Stream and processing streaming data, you'll be having a streaming Dataset.

Being said, the way to write this streaming Dataset is by calling the ForeachWriter, you got it right..

  import org.apache.spark.sql.ForeachWriter
  val writer = new ForeachWriter[Commons.UserEvent] {
  override def open(partitionId: Long, version: Long) = true
  override def process(value: Commons.UserEvent) = {
  processRow(value)
 }
 override def close(errorOrNull: Throwable) = {}
 }

 val query =
 ds.writeStream.queryName("aggregateStructuredStream").outputMode("complete").foreach(writer).start

And the function that writes into topic will be like:

    private def processRow(value: Commons.UserEvent) = {
     /*
     *  Producer.send(topic, data)
     */
   }
2
  • I'm on Java 8, but thank you dude. I'll try later :) May 19, 2017 at 12:50
  • The implementation of this code in Java 8 won't be difficult at all. You'll have a ForeachWriter write = new ForeachWriter(clasz) . The rest override def are private void That's pretty all. I'll leave you to it now May 19, 2017 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.