Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

kafka-consumer-api
Filter by
Sorted by
Tagged with
0 votes
0 answers
1k views

Kafka Consumer is not consuming any messages

I have been trying to configure a Consumer program that should consume or pull the message from Kafka broker but I do not know why the consumer is not consuming any message and not displaying in the ...
Bhaskar Das's user avatar
2 votes
0 answers
1k views

Kafka parsing __consumer_offset data

I can use KafkaTool and kafka-console-consumer to view data from the __consumer_offset topic, but I can't figure out how to parse the data in python if I read it directly with my own custom tool. Even ...
s g's user avatar
  • 5,507
2 votes
1 answer
818 views

Confluent Kafka Connect HDFS Sink connector latency

I am having a kafka topic with 2 million messages and my flush size is 100000 with default partitions in distributed mode with 4 workers and I am able to see the data is written to HDFS immediately in ...
BARATH's user avatar
  • 374
-1 votes
1 answer
187 views

Kafka consumers from two partitions

In kafka I need to consume a topic with two partitions from two consumers (partition 1 to consumer 1 and partition 2 to consumer 2) using Java. This is my Producer Code public class ...
Maravarman Manoharan's user avatar
1 vote
0 answers
665 views

Kafka consumer not consuming messages after restart

Below is the configuration and code for kafka consumer. It works fine on initialization but does not process any further records when consumer restarts after failure. ENABLE_AUTO_COMMIT is false, ...
abhinav singh's user avatar
-1 votes
1 answer
649 views

Generic way to consume data from kafka topics

Is there is way to get list of all kafka topics and consume all data from it gererically using kafka. We actually want to capture any kind of data flowing inside kafka .
sundaram FRM's user avatar
1 vote
2 answers
2k views

Kafka-Python consume last unread message

I have two microservices. MProducer - sending messages to kafka queue MConsumer - reading messages from kafka queue When consumer crashes and restart, I want to continue consuming from last message. ...
Mekan Allaberdiyev's user avatar
1 vote
1 answer
1k views

LoggingMessageFormatter with kafka-avro-console-consumer

I am trying to print avro messages on a kafka topic using kafka-avro-console-consumer in a log4j format. For that I use the following kafka-avro-console-consumer command: bin/kafka-avro-console-...
Anirudha Nanda's user avatar
1 vote
1 answer
8k views

Kafka consumer-client is not registering offset of consumer group on zookeeper

I'm trying to create multiple consumers with different consumer groups to a kafka topic using kafka-clients v.0.10.2.1. Although I'm not able to retrieve the last offset commited by a consumer group. ...
Rodrigo Farias's user avatar
1 vote
0 answers
3k views

TimeBasedPartitioner, Rotate schedule interval, Rotate interval in kafka connect HDFS connector

I would like to flush all the records that are in kafka topic for every hour, Which of this properties can help,So that the data arriving in the topic for an hour is completely available in a file or ...
BARATH's user avatar
  • 374
2 votes
0 answers
535 views

Kafka - Serialization exception; Leading zero error

I am trying to consumer data from KAFKA topic which has data in the following way: { "team_member_id":"0007010267", "accrual_proc_dt":"2018/06/17", "timeoff_plans":"ABC", "length_of_service":357, "...
gklaxman's user avatar
  • 155
1 vote
1 answer
442 views

Kafka Offsets.storage

What is the "offsets.storage" for kafka 0.10.1.1? As per the documentation it shows up under Old Consumer Configs as "zookeeper". offsets.storage zookeeper Select where offsets should be stored (...
Divs's user avatar
  • 1,608
0 votes
1 answer
670 views

Database connection lost when reading from Kafka

I am new to Spring Kafka and I would say Kafka in general. But I am trying to consume a Kafka topic and then populate a database with the events received, so we have this: Kafka Topics -> Application ...
fredlo2020's user avatar
2 votes
3 answers
3k views

Spring boot consumer without using @Kafkalistener

I am trying to write kafka consumer without using @Kafkalistener and below are the lines of code that I am using for configuring listener: @Configuration @EnableKafka public class ...
Rajeev Akotkar's user avatar
1 vote
2 answers
828 views

Kafka : Check if 2 topics have been caught up

I have 2 topics in kafka which I am double writing to. Their offsets are at different commit points to begin with. My consumer starts consuming from the second topic late, but eventually will catch up ...
sharath's user avatar
  • 3,546
-1 votes
1 answer
1k views

How can I check if specific group ID listens to kafka topic?

How can I check if specific group ID listens to kafka topic right now or check amount of group ID that listen to specific topic with any rest API?
alex's user avatar
  • 31
0 votes
1 answer
383 views

Kafka Consumers are balanced across topics

I am creating 6 consumers with same group-id. All the consumers are subscribing to 2 topics with 3 partitions each. Since there are 6 consumers and 6 partitions across 2 topics I am expecting all the ...
Naga's user avatar
  • 33
1 vote
1 answer
1k views

How to perform functional testing of a kafka consumer application?

I want to test an application that consumes Kafka messages and writes them to log. Here is approximate representation of it in scala-like pseudo-code: import kafka.consumer.Consumer import kafka....
Worse_Username's user avatar
3 votes
1 answer
3k views

Kafka consumer group offset goes down to -1

We run a Kafka cluster in Kubernetes based on the gcr.io/google_containers/kubernetes-kafka:1.0-10.2.1 docker image with the zookeeper backend using gcr.io/google_containers/kubernetes-zookeeper:1.0-3....
poppe's user avatar
  • 597
3 votes
1 answer
6k views

Commiting offset after consumer error handling in spring-kafka

I don't fully understand how consumer error handling works with committing offset and akcMode and how it is affected by stopping containers on error (using spring-kafka 1.3.*). Lets say I have two ...
mawek's user avatar
  • 689
2 votes
0 answers
533 views

Error MessageSizeTooLargeError on Apache Kafka

I'm learning Kafka and, while doing a simple exercise, I received the error MessageSizeTooLargeError. The problem is that the message sent by producer to consumer is too big. I tried to change some ...
En1gma's user avatar
  • 29
0 votes
1 answer
2k views

Kafka | Kafka consumer hangs at consumer.poll() | Unexpected error code 2 while fetching data

We are facing an intermittent issue with our Kafka consumer. It gets stuck at consumer.poll(100) method after certain period of time and starts throwing below exception: Unexpected error code 2 ...
user1854373's user avatar
1 vote
1 answer
384 views

Why Multi Kafka Consumer Object makes it always rebalancing, and not work

In the past, I made the wrong decision, that is, operating multiple consumers in a thread (same group id), but there was no corresponding usage scenario at the time, so this problem was not revealed. ...
ZhangJian He's user avatar
1 vote
0 answers
469 views

KafkaConsumer poll() returns partition size of 0 every time after the 1st call

Following the documentation at: https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html I am using Manual Offset Control and am trying a simple test case ...
user10044417's user avatar
3 votes
0 answers
877 views

Kafka Consumer Hung on Certain Partitions - Single Kafka Consumer

In my test environment, I have three Kafka brokers and my topic is having 16 partitions, there are 16 IOT Devices which post the messages to Kafka. I have a single system with one Kafka Consumer ...
Dev Loper's user avatar
  • 219
0 votes
1 answer
531 views

read filter messages from Kafka topic according to consumer

How can we consume custom messages from multiple or a single partition in a Kafka topic? How can I read only 1 message when I consume messages from multiple partitions in a Kafka topic?
Ishant Kumar's user avatar
1 vote
2 answers
1k views

how to specify kafka poll records in specific size?

There's poll method to poll ConsumerRecords , but It can't specify the record's size , how to poll records in specific size ? public ConsumerRecords<K, V> poll(long timeout) {}
shaoyihe's user avatar
  • 1,057
2 votes
1 answer
824 views

Read only last message from Kafka topic [closed]

I am trying to send a video stream through a Kafka topic and doing some processing between each messages consumption. This processing takes approximately 20 milliseconds, so after 100 frames, I have a ...
naifmeh's user avatar
  • 408
0 votes
1 answer
227 views

Before consumers for new topic are attached, I create new topic and produce message in apache kafka

Before consumers for new topic are attached, I create new topic and produce a first message in apache kafka. Then consumers for new topic are attached, but the first message could not be consumed. ...
Eunae Park's user avatar
0 votes
2 answers
490 views

Kafka - read a pack of messages

I have a batch job, which populates data to Kafka topic. Every message has data and job identifier. On the consumer side, I want to only read messages, which belong to this job. After the job has ...
Rahul's user avatar
  • 1,757
2 votes
2 answers
2k views

One Consumer consuming from two different topics on two separate clusters

I have one consumer which I am initializing with a list of brokers (bootstrap.servers). Now in most use cases the list is made of interconnected brokers. So if one connects, then kafka will ...
Sauhard Sharma's user avatar
0 votes
1 answer
4k views

Unable to write to Kafka in appender [Kafka] java.util.concurrent.TimeoutException in Loh4j2

I am trying stream logs from log4j2 to Topic of Kafka. My zookeeper & kafka server are running is running.I have already created a topic for the same. Unable to write to Kafka in appender [Kafka]...
Siva Ram's user avatar
3 votes
2 answers
770 views

How to manage small files created due to insertion of stream data into Hive?

I am reading Kafka messages using simple Kafka consumer. Storing the output into HDFS and doing some filtering. After filtration, I am writing this data into Hive, which causes small orc files into ...
Renu's user avatar
  • 140
1 vote
0 answers
990 views

Why is kafka poll method blocking?

Writing to Kafka is non-blocking but reading from it blocks thread. Why does the Kafka poll method not use non-blocking API and then receive records in a callback?
Amir Hossein Javan's user avatar
0 votes
1 answer
645 views

Trigger resp api on kafka event

we are creating some kind of a business process pipeline using wso2 and kafka. Do you know if it's possible to trigger some rest API on kafka event? Thanks,
Alex's user avatar
  • 1,960
2 votes
1 answer
5k views

Not able consume messages from remote machine in Kafka

I have created a kafka topic in one of my machine having IP 192.168.25.50 of topic name test-poc. Then by using kafka-console-producer i have produced message as like below kafka-console-producer --...
Priyesh Karatha's user avatar
0 votes
0 answers
549 views

How to reload/re-initialize Kafka Consumer with new topic list added to DB

I have this code that loads when my node back end starts, I just changed this code from using a static list of topics for the Kafka Consumer and it now uses an array of topics it reads from the ...
Baub's user avatar
  • 741
1 vote
3 answers
1k views

Re-consume messages for which offset was not commited

I have a custom Kafka Consumer in which I use to send some requests to a REST API. According to the response from the API, I either commit the offset or skip the message without commit. Minimal ...
Giorgos Myrianthous's user avatar
5 votes
2 answers
20k views

Kafka Consumer Error

I am using a kafka produer and a Spring kafka consumer. I am using a Json serializer and deserializer. Whenever I try to read messages in the consumer from the topic i get the following error: org....
jenjen's user avatar
  • 163
2 votes
1 answer
2k views

Error in kafka consumer while writing data from input topic to output topic using kafka streams

By using kafka connector i'm writing the avro formatted data into kafka topic and then by using kafka streams i'm mapping some values and writing the output to some other topic by using: Stream.to(&...
Sujitha Chinnu's user avatar
2 votes
1 answer
7k views

Missing required argument "[zookeeper]"

i'm trying to start a consumer using Apache Kafka, it used to work well, but i had to format my pc and reinstall everything again, and now when trying to run this: bin/kafka-console-consumer.sh --...
Miguel 2488's user avatar
  • 1,420
2 votes
1 answer
921 views

How to read messages after the last consumed message in case of consumer shutdown in Kafka?

I have a way of performing this task in shell: How to make kafka consumer to read from last consumed offset but not from beginning However, I am willing to do this in Python, using kafka-python I ...
hasherBaba's user avatar
2 votes
2 answers
3k views

Occasionally Messages posted to the kafka went missing

Hi I am having a custom framework, which exposes API to publish as well as consume. I support a lot of topics for a user to post to and consume. This seems to work fine except that occasionally, after ...
Suresh C Nair's user avatar
0 votes
1 answer
156 views

what will be the behaviour of Kafka...?

I am stuck in a typical use case or scenario where I am not sure what will be the behavior of Kafka.. SCENERIO : I am using Spring Kafka with spring Boot. In my application I am having one Rest end ...
adhikari's user avatar
  • 112
1 vote
0 answers
261 views

ConsumerGroup consumerOptions Configuration error @types/kafka-node

I'm trying to use consumerGroup instead of HighLevelConsumer in a typescript project, but I'm not able to configure consumerOptions for consumerGroup. I am getting the following error when I assign ...
SathyaJ's user avatar
  • 11
6 votes
4 answers
19k views

How to read messages from kafka consumer group without consuming?

I'm managing a kafka queue using a common consumer group across multiple machines. Now I also need to show the current content of the queue. How do I read only those messages within the group which ...
Priyam Singh's user avatar
0 votes
1 answer
957 views

Not able to consume messages using kafka-console-consumer.sh

I have set up an AWS cluster of 3 nodes.I have modified the /etc/hosts file for the nodes which looks something like this 172.31.9.2 kafka1 172.31.9.2 zookeeper1 172.31.19.230 kafka2 172.31.19.230 ...
Nagaraj B's user avatar
1 vote
1 answer
4k views

Spring Kafka Consumer Sometime stop receive messages

Who can help me with Kafka Consumer what in sometime stop receive messages from topic. I use Spring Boot 2.0.2 Finchley.RC2 and Spring Kafka 2.1.7.RELEASE. Kafka Version 1.0.1 Kafka version : 1.0....
Fonexn's user avatar
  • 301
1 vote
1 answer
342 views

How to save only last message from topic to a file?

As in the title - how to save only last message from topic to a file? I've tried to do this but when I open a file it contains about last 10 message that I've send before. Here is my code: public ...
asketoo's user avatar
  • 11
0 votes
2 answers
4k views

Why producer.send(record).get() works but producer.send(record, callback) does not work

I'm using kafka to send messages to a consumer. But for some reason when I send a record into a topic with producer.send(record, new MyProducerCallback()); the consumer for that topic does not receive ...
cod3min3's user avatar
  • 595

1
56 57
58
59 60
87