I would like to have a consumer actor subscribe to a Kafka topic and stream data for further processing with Spark Streaming outside the consumer. Why an actor? Because I read that its supervisor strategy would be a great way to handle Kafka failures (e.g., restart on a failure).
I found two options:
- The Java
KafkaConsumer
class: itspoll()
method returns aMap[String, Object]
. I would like aDStream
to be returned just likeKafkaUtils.createDirectStream
would, and I don't know how to fetch the stream from outside the actor. - Extend the
ActorHelper
trait and useactorStream()
like shown in this example. This latter option doesn't display a connection to a topic but to a socket.
Could anyone point me in the right direction?