I have spark streaming application which consumes kafka messages. And I want to process all messages coming last 10 minutes together. Looks like there are two approaches to do job done:
val ssc = new StreamingContext(new SparkConf(), Minutes(10))
val dstream = ....
and
val ssc = new StreamingContext(new SparkConf(), Seconds(1))
val dstream = ....
dstream.window(Minutes(10), Minutes(10))
and I just want to clarify is there any performance differences between them
batchDuration
inStreamingContext
defines the size of the microbatch.window()
depends on your application