2

my spark streaming version is 2.0, the kafka version is 0.10.0.1,spark-streaming-kafka-0-10_2.11. I use the direct way to get kafka records,I now want to limit the maximum number of messages I get in a batch. so I set the max.poll.records value,but it does not work. The number of consumers in spark is the number of partitions in kafka?so the maximum number of records in spark streaming is max.poll.records*consumers?

1
  • That property is an upper bound, not an exact number. Also, not sure what you're asking about consumers, but how many executors do you have? Sep 27, 2018 at 3:00

1 Answer 1

3

max.poll.records controls the upper bound for the number of records returned from the poll.

In spark streaming more than one poll might happen in one batch. In which case max.poll.records will not be very useful. You should use spark.streaming.kafka.maxRatePerPartition, according to documentation

An important one is spark.streaming.kafka.maxRatePerPartition which is the maximum rate (in messages per second) at which each Kafka partition will be read by this direct API

So the the max number of records per batch will be

(spark.streaming.kafka.maxRatePerPartition) * (batch duration in seconds) * (number of kafka partitions)

e.g if you have 2 partitions in the topic, batch duration is 30 seconds and spark.streaming.kafka.maxRatePerPartition is 1000 you would see 6000 (2 * 30 * 1000) records per batch.

It might be useful to also enable spark.streaming.backpressure.enabled to have a more adaptive rate based on the time taken to process a batch.

More info about under the hood working of kafka direct stream

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.