2

For some larger messages I encountered the following error:

kafka.common.MessageSizeTooLargeException: Message size is 1185198 bytes which exceeds the maximum configured message size of 1000012.

So as per this thread increased the message size at broker and consumer:

fetch.message.max.bytes=10485760
replica.fetch.max.bytes=10485760
message.max.bytes=10485760


added to config/server.properties

But then the messages goes through but consumer error out:

[2015-08-26 21:08:08,722] ERROR Error processing message, stopping consumer:  (kafka.tools.ConsoleConsumer$)
kafka.common.MessageSizeTooLargeException: Found a message larger than the maximum fetch size of this consumer on topic xyz partition 0 at fetch offset 29. Increase the fetch size, or decrease the maximum message size the broker will allow.
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:90)
    at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
    at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
    at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
    at scala.collection.Iterator$class.foreach(Iterator.scala:660)
    at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
    at kafka.consumer.KafkaStream.foreach(KafkaStream.scala:25)
    at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:166)
    at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Consumed 3 messages

It seems the consumer is not picking up fetch.message.max.bytes=10485760

kafka_2.9.1-0.8.2.1

Any pointers?

1 Answer 1

5

You shouldn't put fetch.message.max.bytes to config/server.properties but to your ConsumerConfig (see here for details). If you are using console consumer you may pass a --consumer.config consumer.properties flag where consumer.properties file will contain this config value.

5
  • How can I get Storm spout to obey the higher fetch size. Any idea? @serejja Aug 28, 2015 at 15:11
  • @KedarParikh no idea, never used Storm that deep before
    – serejja
    Aug 29, 2015 at 11:27
  • SpoutConfig.fetchSizeBytes seems to be the option Sep 10, 2015 at 16:14
  • Passing --consumer.config consumer.properties works but when I tried to put that file in the same folder where server.properties lies, it does not work. Did i miss anything?
    – Sujan
    Oct 31, 2016 at 17:41
  • @Sujan probably because console consumer does not scan config directory for consumer properties
    – serejja
    Nov 1, 2016 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.