Questions tagged [apache-kafka-connect]
Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.
3,710
questions
0
votes
0
answers
16
views
Kafka Mirror to Apache Kafka from AWS MKS Cluster
I'm confused about the architecture of mirroring a topic among an AWS MKS cluster (source) to an Apache Kafka 2.13 cluster (target).
The source cluster also uses AWS Glue Schema registry, so I need to ...
-2
votes
0
answers
31
views
kafka for pure javascript without nodejs [closed]
I'm working on a project where I need to consumer topic from an Apache Kafka topic using pure JavaScript.(Without Node) Is there a way to achieve this ?
Additionally, any tips or best practices for ...
0
votes
1
answer
22
views
I am facing slowness in getting streaming data while working with kafka
I am facing lags while working with kafka streaming, there is no lag or slowness while it is snapshot ting data but when it starts its streaming I am facing lags which is starting from 10 mins and it ...
0
votes
0
answers
29
views
Kafka Sink Connector - Custom Transformer: Connect to Azure Key Vault
I have this connector that takes the information to a non-relational Database. What I want is that before I reach said component (DB) I want to perform certain transformations and for that I need to ...
0
votes
1
answer
11
views
Kafka Connect Dynamic Routing without Kafka Stream
I am working on a Integration layer, where I want to use Kafka Connect between Kafka Topics to achieve dynamic routing.
I want to know if it is possible to use Kafka Connect to connect between 2 ...
0
votes
1
answer
24
views
kafka jdbc connector startup failed due to dialect recognition error
I am trying to start kafka jdbc sink connector
https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc
to save data from topic to postgreSql database. I am using the following configuration
{
&...
0
votes
1
answer
14
views
how do I set key as null in when using sftp csv connector in kafka. I need this to distribute the records in round robin manner across brokers
This is the configuration I am using for my csv source connector. This is producing key as "{}" an empty struct. I think this is causing all the messages to accumulate in one partition.
&...
0
votes
0
answers
11
views
Connection Error for SQL Server Debezium for Apache Kafka
I am trying to set up CDC on my SQL Server and send messages to Apache Kafka. I am getting the following error:
The 'database.names' value is invalid: Cannot be empty.
You can also find the above list ...
0
votes
0
answers
19
views
kafka http sink connector regex.patterns/regex.replacements
I am enabling an http sink connector with multipe kafka topics. I would like to use kafka's templating url feature with regex find and replace
the url form would be as follows
example.com/{$topic}
...
1
vote
0
answers
30
views
Snowflake Sink connector: Connect Confluent and Snowflake Across Different Clouds and Regions
I want to connect my Confluent Cloud with Snowflake using Snowflake Sink connector. While I managed to do that when both Confluent cluster and Snowflake account are in the same provider and same ...
0
votes
0
answers
16
views
Testing custom Kafka Connect SMT using Test Containers
I'm trying to use TestContainers to integration test a Kafka Connect Custom SMT (Simple Message Transformer) which needs to make use of a external library (BouncyCastle encryption).
The test should:
...
0
votes
0
answers
17
views
Forcing Debezium SQL Server connector to create topic for empty table
I am using a Debezium source connector to ingest data from tables under a SQL Server instance, and them publishing the data to a Kafka instance (and the respective schemas to a Schema Registry).
The ...
0
votes
1
answer
42
views
How to disable automatic registration of schema on the Azure Schema Registry in Kafka Connect scenario?
Does anyone know whether it is possible to setup the source connector in Kafka Connect scenarios so that it does not automatically register schemas on the Azure Schema Registry?
I ask since I'm ...
0
votes
1
answer
31
views
Start Kafka connector using the source code
I want to start the Kafka connector using the source code, but I always encounter some errors.
this is my configuration.I don't know if there's something wrong with the way I configured it.
...
0
votes
0
answers
22
views
Tombstone values not written in Cassandra
I have a sink connector that is writing a JSON message to a Cassandra table. The Cassandra table has a row with some columns having timestamps values. A new message comes to the sink connector and now ...
0
votes
0
answers
23
views
Kafka Connect Worker Fails to Update Session Key Until Exactly One Hour Later - How to Configure Retries?
I'm working with Kafka Connect 3.4.0 and I've encountered an issue that I'm struggling to understand. I have logs that show the Kafka Connect worker failing to read a session key from the config topic ...
0
votes
0
answers
29
views
Kafka S3 source connector does not include key to the messages with SMT
I have set up kafka s3 source connector that reads messages that have been inserted to the bucket by s3 sink connector. I noticed that source connector does not include by default the key in the ...
0
votes
2
answers
42
views
JDBC parameter to avoid ORA-22922 : nonexistant LOB value
I am working on a kafka connect that get information from a database using an sql request.
This sql request is like a
select
X,X,X,
JSON_ARRAYAGG(
JSON_OBJECT(
select X,X,...
0
votes
0
answers
9
views
Is there a way to identify whether dataloss is present while consuming data from Kafka Topic
Wanted to know is there any way to identify the dataloss from the consumer while consuming data from the Kafka Topic.
Producer has sent message to the Topic but consumer is not received.
For example ...
0
votes
0
answers
29
views
Kafka Sink Connector taking time to run for the first time
I created the source and the sink connectors below to synchronize two mysql databases, but after uploading the two connectors and perform an insertion in the source database for the first time, it ...
0
votes
0
answers
7
views
How to read properties from kafka connect configuration in custom SMT?
I have setup debezium mysql source connector to capture mysql change data. I have also implemented a Custom Filter using SMT custom transformation.
I want to access some configs from the connector ...
0
votes
0
answers
5
views
Apache Kafka Connector: Where to put global resources
Suppose I have something like a PoolingHttpClientConnectionManager or ActiveMQConnectionFactory that I need to use with my connector. I do not want to create one of these per task. Where do I place ...
0
votes
1
answer
28
views
ksqlDB fails to deserialize JSON key
I'm trying to load data into a ksqlDB source table by using the JDBC Source connector in Kafka Connect. The topic key is a JSON object created from two of the columns. ksqlDB can't deserialize the key ...
0
votes
1
answer
13
views
Can you load elements in a nested json into multiple rows in Big Query from Kafka using Kafka Connect and Big Query Sink?
I'm looking to see if Kafka Connect is a possible solution for my use case. I need to write from Kafka into a Big Query Dataset in near real time. My message contains multiple nested elements which ...
0
votes
2
answers
85
views
kafka producer WARNING: NOT_ENOUGH_REPLICAS Got error produce response with correlation id on topic-partition mytopic-events-0, retrying
We are seeing this warning 10 times a second in debezium-mysql-connector logs:
[kafka-producer-network-thread | connect-distributed-offsets]
WARN org.apache.kafka.clients.producer.internals.Sender - ...
0
votes
0
answers
13
views
Kafka Connect: influence assignment of tasks to workers
We run a small Kafka Connect cluster (or rather, MirrorMaker 2 in distributed mode, which uses Kafka Connect in the background). All tasks in this cluster are configured with tasks.max greater than 1. ...
-1
votes
0
answers
9
views
Debezium 2.x not discovering mongodb plugin
I'm trying to use debezium to stream mongodb changes in kafka using kafka connect, but with version 2.x of debezium mongodb connector it is not discovering the mongodb plugin given under path plugin....
0
votes
0
answers
7
views
Kafka connect - improving failover time when using with debezium
Using debezium/connect which seems to be just a fancy wrapper around kafka-connect distributed version and uses bin/connect-distributed.sh
Docker compose config for testing:
debezium-connect-0:
...
0
votes
0
answers
27
views
Error decoding/mapping Kafka record: CDC Pipeline from PostgreSQL to Cassandra DB?
I am encountering an issue while working with Change Data Capture using Kafka Connect.
Problem:
I am working on a Change Data Capture Pipeline, and I'm getting the following error message when trying ...
-1
votes
2
answers
61
views
StreamSource: ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:84)
so I am trying to run the connect-standalone.sh to produce the data from log file as messages into Kafka topic by passing both the properties files
INFO Kafka Connect started (org.apache.kafka.connect....
0
votes
0
answers
11
views
Kafka Connect in timestamp mode never stops reading
I am building a Kafka Connector to a SQL Server database in timestamp mode and it's already reading and transforming data but it never stops reading. Once it reaches the last row of the table it ...
0
votes
1
answer
21
views
Strimzi kafka connect cannot find my custom transformer but finds connector
I am running a Strimzi KafkaConnect cluster which loads a custom transformer to apply a SMT to our messages. The .jar is within the image and loaded by Strimzi KafkaConnect cluster (appears in logs):
...
0
votes
0
answers
36
views
io.debezium.data.VariableScaleDecimal (STRUCT) type doesn't have a mapping to the SQL database column type
I'm trying to stream data from Oracle to Postgres with Debezium as source and JdbcSinkConnector by Confluent as sink, and on Sink stage I'm getting this error:
Caused by: org.apache.kafka.connect....
0
votes
0
answers
19
views
Kafka connect mirror maker 2.0 how to replicate data between two postgres databases
I am trying to replicate data between two Postgres databases on separate k3s clusters, each on their own machine, in a one-way data flow. I have been struggling to get my topics replicated from server ...
0
votes
1
answer
44
views
kafka-connect-bigquery connector failure
I am utilizing wepay/kakfa-connect-bigquery:2.5.0 and replicating tables using it to BigQuery. Frequently for some topic, an error is encountered while writing to BigQuery and after that the data ...
0
votes
0
answers
43
views
Kafka Oracle Target via JDBC Sink connector getting unwanted data in the PK column in JSON format. How to strip?
I am using debezium connector to pull data from Oracle and JDBC Sink connector to insert into Oracle.
I am seeing more data in the primary column for target. Any idea how to strip the data in primary ...
1
vote
1
answer
43
views
How to run a Kafka-Connect SInk in standalone mode
I would like to run a kafka connect sink in standalone mode for testing purposes, because in spite of having a beefy computer I don't want to overwork it unnecessarily...
I have this configuration in ...
0
votes
0
answers
33
views
Kafka to Database loader - Database having stale data when querying
We are building a real time collaboration form (web app). The front-end and the back-end are connected through web socket.The user-1 and user-2 are in the same form (say). The changes they make will ...
0
votes
1
answer
41
views
How could Kafka Connect help persisting Avro to a database and why is this a good design approach?
According to this stack Kafka Connect would help prevent performing a "2-step commit" when publishing to Kafka and persisting the avro message to a database, and that it follows a "read ...
0
votes
0
answers
44
views
Kafka Debezium connect to Oracle
So, I have Oracle DB 19c and try to connect to Kafka (with Debezium) on another server.
I already have DB, connector - which one dont work correctly
{
"name": "ag-test3",
&...
0
votes
0
answers
20
views
Confluent Kafka S3 source connector - not a valid AVRO format
I am trying to setup a Confluent kafka s3 source connector to read from S3 and add messages to Kafka topic. I am struggling with the formats - it asks for input and output format. What those mean? I ...
0
votes
0
answers
39
views
Unable to initialize REST resources Error when trying to run kafka connect, connecting to Azura data storage sink on docker
I'm running docker with zookeeper, kafka, kafka-connect and azure-storage-sink.
zookeeper and kafka run successfully but kafka-connect and azure-storage-sink containers exit with this error:
[2023-08-...
0
votes
0
answers
48
views
Kafka/Snowflake Connector: "snowflake.schema.name schema does not exist"
I try to use the Snowflake Connector (snowflake-kafka-connector-2.0.0.jar
) in my Kafka (kafka_2.13-3.5.0). But when starting the connector, I get: "snowflake.schema.name schema does not exist&...
0
votes
0
answers
40
views
Debezium source connector database migrations breaking AVRO schema compatibility
Sorry for the stupid title.
We're using Debezium source connector to stream changes from PostgreSQL to kafka. Along the way the connector is registering schemas to confluent schema-registry.
Streaming ...
0
votes
1
answer
86
views
Kafka connect Graceful stop of task failed
Context: using kafka connect 7.4.0 to read data from Postgres views and write to another Postgres tables.
All works good, but whenever I stop the source connectors via the rest api, I always get:
...
0
votes
0
answers
53
views
Migrate Kafka cluster to Azure Eventhubs
My objective is to migrate data from a kafka cluster to Azure eventhub . I tried MirrorMaker2 , it works but it requires namespace level admin privileges to work as MirrorMaker creates some internal ...
0
votes
0
answers
25
views
Kafka connect is not consuming data continuously
As shown in the attached diagram Kafka Connect is consuming data only every 5 mins, any idea how to fix this and ensure continuous consumption.
0
votes
0
answers
28
views
Kafka Connector - why is consumer creating a topic even though connector is not existing?
Im confused at this subject. One of my services has a consumer which listens on a topic. Im checking the topics in CMAK. If my consumer is up and running and if I delete the topic in CMAK it instantly ...
0
votes
1
answer
41
views
Failed to deserialize data for topic "testtopic" to Avro for sink connector
Here is the sink connector properties
"name": "sink-testtopic-crdb",
"config": {
"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector",
&...
1
vote
2
answers
40
views
Is there any way to get the bad record which is causing the issue while sending it to the topic using source connector?
In sink connector(I am using Jdbc Sink Connector), if I use the following property I will get the bad or error causing record in the mentioned topic(i,e dead_topic).
"errors.tolerance": &...