0

I've implemented a heron topology which reads messages from Kafka queue. Hence my topology has a kafka spout and a bolt which counts the number of messages read from the queue.

When I emit say 10000 messages into kafka queue, I can see all the messages being received in the kafka spout in heron topology however there is few messages being lost at the bolt.

Following is the topology settings for heron

 Config config = Config.newBuilder()
                    .setUserConfig("topology.max.spout.pending", 100000)

                    .setUserConfig("topology.message.timeout.secs", 100000)
                    .setNumContainers(1)
                    .setPerContainerCpu(3)
                    .setPerContainerRamInGigabytes(4)
                    .setDeliverySemantics("ATLEAST_ONCE")
                    .build();

Any pointers would be helpful.

EDIT: I'm using streamlet API of heron. I replaced the count bolt with the log bolt but seeing the same issue of message drop in the logs of log bolt

processingGraphBuilder.newSource(kafkaSource)
                      .log();

EDIT 2: I resolved the issue by completely removing the streamlet API. I reimplemented everything using basic spout and bolt API and had acking at the spout. This fixed the issue. I guess this happened due to no acking happening at spout in streamlet API

2 Answers 2

1

Simple answer: shouldn't drop.

A few questions: - in heronui, what are the all time emit and ack counts of your spout? - in heronui, what are the all time execute, ack and failure counts of your bolt?

4
  • I wrote 1M messages into kafka queue from a producer and I see all the messages being read by the kafka spout in heron topology. I determined that the messages are received by the kafka spout by checking the heron logs. However in heron-ui I see only 567188 emit count and 0 as ack count.
    – tourist
    Nov 8, 2018 at 16:54
  • I see all the 1M messages being logged in heron logs at spout. However I see few messages being missed in heron bolt which performs count.
    – tourist
    Nov 8, 2018 at 16:57
  • Sorry I misread the logs in heron-ui. The ack count is same as emit count at bolt. However the ack count at spout is 0 where as the emit count is same as number of messages
    – tourist
    Nov 8, 2018 at 19:18
  • If kafka has 1M messages and each message is converted to a tuple, spout emit count should be exactly 1M. Otherwise the spout might need some investigation. The bolt execution count should be the same as spout emit count, so it should be 1M as well. These numbers should be helpful for you to locate the issue. On the other hand, the ack count should be the same as the emit count if your topology is in atleast_once mode. 0 is expected if the topology is running in atmost_once mode.
    – Ning Wang
    Nov 9, 2018 at 17:07
1

When you say that messages are dropped are you seeing fails registered in the fail count metric or just that your execute count in the bolt does not tally with the emit count of the spout?

In Storm-compatibility mode metrics are calculated based on a sample (default is 5% I think). Therefore counts could be out by that margin. For example, depending on when the stream is sampled, you could send 100 tuples through and the execute count could be 80 or 120.

5
  • I'm seeing drop at the execute count bolt in the topology. Did I miss some configuration? Is this normal to see mismatch between spout and bolt?
    – tourist
    Nov 7, 2018 at 13:55
  • Config looks ok to me.
    – Ning Wang
    Nov 7, 2018 at 18:14
  • The emit count of spout should meet the execute count of the bolt. These metrics are available in the Heron UI. You can do a quick check. Also, could you clarify the detailed symptoms of "message being lost"?
    – Neng
    Nov 7, 2018 at 21:35
  • @Neng, I checked in heron-ui and I see only 567188 as emit count and 0 as ack count. The messages are just some temperature readings read from kafka queue.
    – tourist
    Nov 8, 2018 at 16:55
  • @Neng, In terms of the symptoms of the messages being lost, I see the last few messages being dropped at bolt. However those messages are being logged at spout logs. For instance if I send 1M messages into kafka, 1M are received at kafka spout but first 996839 messages are received by bolt but later messages aren't received.
    – tourist
    Nov 8, 2018 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.