Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Kafka Mirror to Apache Kafka from AWS MKS Cluster

I'm confused about the architecture of mirroring a topic among an AWS MKS cluster (source) to an Apache Kafka 2.13 cluster (target). The source cluster also uses AWS Glue Schema registry, so I need to ...
Siraxis's user avatar
  • 15
-2 votes
0 answers
31 views

kafka for pure javascript without nodejs [closed]

I'm working on a project where I need to consumer topic from an Apache Kafka topic using pure JavaScript.(Without Node) Is there a way to achieve this ? Additionally, any tips or best practices for ...
lpdevlop's user avatar
0 votes
1 answer
22 views

I am facing slowness in getting streaming data while working with kafka

I am facing lags while working with kafka streaming, there is no lag or slowness while it is snapshot ting data but when it starts its streaming I am facing lags which is starting from 10 mins and it ...
Muhammad Affan's user avatar
0 votes
0 answers
29 views

Kafka Sink Connector - Custom Transformer: Connect to Azure Key Vault

I have this connector that takes the information to a non-relational Database. What I want is that before I reach said component (DB) I want to perform certain transformations and for that I need to ...
DarkVaderM's user avatar
0 votes
1 answer
11 views

Kafka Connect Dynamic Routing without Kafka Stream

I am working on a Integration layer, where I want to use Kafka Connect between Kafka Topics to achieve dynamic routing. I want to know if it is possible to use Kafka Connect to connect between 2 ...
Adnan Mian's user avatar
0 votes
1 answer
24 views

kafka jdbc connector startup failed due to dialect recognition error

I am trying to start kafka jdbc sink connector https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc to save data from topic to postgreSql database. I am using the following configuration { &...
papaj's user avatar
  • 1
0 votes
1 answer
14 views

how do I set key as null in when using sftp csv connector in kafka. I need this to distribute the records in round robin manner across brokers

This is the configuration I am using for my csv source connector. This is producing key as "{}" an empty struct. I think this is causing all the messages to accumulate in one partition. &...
NOOBCODER's user avatar
0 votes
0 answers
11 views

Connection Error for SQL Server Debezium for Apache Kafka

I am trying to set up CDC on my SQL Server and send messages to Apache Kafka. I am getting the following error: The 'database.names' value is invalid: Cannot be empty. You can also find the above list ...
btava001's user avatar
  • 131
0 votes
0 answers
19 views

kafka http sink connector regex.patterns/regex.replacements

I am enabling an http sink connector with multipe kafka topics. I would like to use kafka's templating url feature with regex find and replace the url form would be as follows example.com/{$topic} ...
tomsoyer's user avatar
1 vote
0 answers
30 views

Snowflake Sink connector: Connect Confluent and Snowflake Across Different Clouds and Regions

I want to connect my Confluent Cloud with Snowflake using Snowflake Sink connector. While I managed to do that when both Confluent cluster and Snowflake account are in the same provider and same ...
Evangelos Malandrakis's user avatar
0 votes
0 answers
16 views

Testing custom Kafka Connect SMT using Test Containers

I'm trying to use TestContainers to integration test a Kafka Connect Custom SMT (Simple Message Transformer) which needs to make use of a external library (BouncyCastle encryption). The test should: ...
bobmarksie's user avatar
  • 3,282
0 votes
0 answers
17 views

Forcing Debezium SQL Server connector to create topic for empty table

I am using a Debezium source connector to ingest data from tables under a SQL Server instance, and them publishing the data to a Kafka instance (and the respective schemas to a Schema Registry). The ...
Marco Moraes's user avatar
0 votes
1 answer
42 views

How to disable automatic registration of schema on the Azure Schema Registry in Kafka Connect scenario?

Does anyone know whether it is possible to setup the source connector in Kafka Connect scenarios so that it does not automatically register schemas on the Azure Schema Registry? I ask since I'm ...
Longclaw's user avatar
0 votes
1 answer
31 views

Start Kafka connector using the source code

I want to start the Kafka connector using the source code, but I always encounter some errors. this is my configuration.I don't know if there's something wrong with the way I configured it. ...
Red256's user avatar
  • 1
0 votes
0 answers
22 views

Tombstone values not written in Cassandra

I have a sink connector that is writing a JSON message to a Cassandra table. The Cassandra table has a row with some columns having timestamps values. A new message comes to the sink connector and now ...
Prashanth Sirikonda's user avatar
0 votes
0 answers
23 views

Kafka Connect Worker Fails to Update Session Key Until Exactly One Hour Later - How to Configure Retries?

I'm working with Kafka Connect 3.4.0 and I've encountered an issue that I'm struggling to understand. I have logs that show the Kafka Connect worker failing to read a session key from the config topic ...
Andrei Pietrusel's user avatar
0 votes
0 answers
29 views

Kafka S3 source connector does not include key to the messages with SMT

I have set up kafka s3 source connector that reads messages that have been inserted to the bucket by s3 sink connector. I noticed that source connector does not include by default the key in the ...
martin.bolco's user avatar
0 votes
2 answers
42 views

JDBC parameter to avoid ORA-22922 : nonexistant LOB value

I am working on a kafka connect that get information from a database using an sql request. This sql request is like a select X,X,X, JSON_ARRAYAGG( JSON_OBJECT( select X,X,...
Hyry's user avatar
  • 1
0 votes
0 answers
9 views

Is there a way to identify whether dataloss is present while consuming data from Kafka Topic

Wanted to know is there any way to identify the dataloss from the consumer while consuming data from the Kafka Topic. Producer has sent message to the Topic but consumer is not received. For example ...
John's user avatar
  • 1
0 votes
0 answers
29 views

Kafka Sink Connector taking time to run for the first time

I created the source and the sink connectors below to synchronize two mysql databases, but after uploading the two connectors and perform an insertion in the source database for the first time, it ...
sammubr's user avatar
  • 545
0 votes
0 answers
7 views

How to read properties from kafka connect configuration in custom SMT?

I have setup debezium mysql source connector to capture mysql change data. I have also implemented a Custom Filter using SMT custom transformation. I want to access some configs from the connector ...
vinay's user avatar
  • 49
0 votes
0 answers
5 views

Apache Kafka Connector: Where to put global resources

Suppose I have something like a PoolingHttpClientConnectionManager or ActiveMQConnectionFactory that I need to use with my connector. I do not want to create one of these per task. Where do I place ...
JavaStudent's user avatar
0 votes
1 answer
28 views

ksqlDB fails to deserialize JSON key

I'm trying to load data into a ksqlDB source table by using the JDBC Source connector in Kafka Connect. The topic key is a JSON object created from two of the columns. ksqlDB can't deserialize the key ...
jerryman's user avatar
0 votes
1 answer
13 views

Can you load elements in a nested json into multiple rows in Big Query from Kafka using Kafka Connect and Big Query Sink?

I'm looking to see if Kafka Connect is a possible solution for my use case. I need to write from Kafka into a Big Query Dataset in near real time. My message contains multiple nested elements which ...
DPEZ's user avatar
  • 107
0 votes
2 answers
85 views

kafka producer WARNING: NOT_ENOUGH_REPLICAS Got error produce response with correlation id on topic-partition mytopic-events-0, retrying

We are seeing this warning 10 times a second in debezium-mysql-connector logs: [kafka-producer-network-thread | connect-distributed-offsets] WARN org.apache.kafka.clients.producer.internals.Sender - ...
user674669's user avatar
  • 10.8k
0 votes
0 answers
13 views

Kafka Connect: influence assignment of tasks to workers

We run a small Kafka Connect cluster (or rather, MirrorMaker 2 in distributed mode, which uses Kafka Connect in the background). All tasks in this cluster are configured with tasks.max greater than 1. ...
arov00's user avatar
  • 169
-1 votes
0 answers
9 views

Debezium 2.x not discovering mongodb plugin

I'm trying to use debezium to stream mongodb changes in kafka using kafka connect, but with version 2.x of debezium mongodb connector it is not discovering the mongodb plugin given under path plugin....
Krishan's user avatar
0 votes
0 answers
7 views

Kafka connect - improving failover time when using with debezium

Using debezium/connect which seems to be just a fancy wrapper around kafka-connect distributed version and uses bin/connect-distributed.sh Docker compose config for testing: debezium-connect-0: ...
gerasalus's user avatar
  • 7,558
0 votes
0 answers
27 views

Error decoding/mapping Kafka record: CDC Pipeline from PostgreSQL to Cassandra DB?

I am encountering an issue while working with Change Data Capture using Kafka Connect. Problem: I am working on a Change Data Capture Pipeline, and I'm getting the following error message when trying ...
Jasir Arfat's user avatar
-1 votes
2 answers
61 views

StreamSource: ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:84)

so I am trying to run the connect-standalone.sh to produce the data from log file as messages into Kafka topic by passing both the properties files INFO Kafka Connect started (org.apache.kafka.connect....
Oyindamola Victor's user avatar
0 votes
0 answers
11 views

Kafka Connect in timestamp mode never stops reading

I am building a Kafka Connector to a SQL Server database in timestamp mode and it's already reading and transforming data but it never stops reading. Once it reaches the last row of the table it ...
Misael Alarcon's user avatar
0 votes
1 answer
21 views

Strimzi kafka connect cannot find my custom transformer but finds connector

I am running a Strimzi KafkaConnect cluster which loads a custom transformer to apply a SMT to our messages. The .jar is within the image and loaded by Strimzi KafkaConnect cluster (appears in logs): ...
James F's user avatar
  • 11
0 votes
0 answers
36 views

io.debezium.data.VariableScaleDecimal (STRUCT) type doesn't have a mapping to the SQL database column type

I'm trying to stream data from Oracle to Postgres with Debezium as source and JdbcSinkConnector by Confluent as sink, and on Sink stage I'm getting this error: Caused by: org.apache.kafka.connect....
Yevgeniy's user avatar
0 votes
0 answers
19 views

Kafka connect mirror maker 2.0 how to replicate data between two postgres databases

I am trying to replicate data between two Postgres databases on separate k3s clusters, each on their own machine, in a one-way data flow. I have been struggling to get my topics replicated from server ...
Patrick Roche's user avatar
0 votes
1 answer
44 views

kafka-connect-bigquery connector failure

I am utilizing wepay/kakfa-connect-bigquery:2.5.0 and replicating tables using it to BigQuery. Frequently for some topic, an error is encountered while writing to BigQuery and after that the data ...
Raman's user avatar
  • 1,231
0 votes
0 answers
43 views

Kafka Oracle Target via JDBC Sink connector getting unwanted data in the PK column in JSON format. How to strip?

I am using debezium connector to pull data from Oracle and JDBC Sink connector to insert into Oracle. I am seeing more data in the primary column for target. Any idea how to strip the data in primary ...
user3597043's user avatar
1 vote
1 answer
43 views

How to run a Kafka-Connect SInk in standalone mode

I would like to run a kafka connect sink in standalone mode for testing purposes, because in spite of having a beefy computer I don't want to overwork it unnecessarily... I have this configuration in ...
ETisREAL's user avatar
0 votes
0 answers
33 views

Kafka to Database loader - Database having stale data when querying

We are building a real time collaboration form (web app). The front-end and the back-end are connected through web socket.The user-1 and user-2 are in the same form (say). The changes they make will ...
SB Praveen's user avatar
0 votes
1 answer
41 views

How could Kafka Connect help persisting Avro to a database and why is this a good design approach?

According to this stack Kafka Connect would help prevent performing a "2-step commit" when publishing to Kafka and persisting the avro message to a database, and that it follows a "read ...
Hercules Konsoulas's user avatar
0 votes
0 answers
44 views

Kafka Debezium connect to Oracle

So, I have Oracle DB 19c and try to connect to Kafka (with Debezium) on another server. I already have DB, connector - which one dont work correctly { "name": "ag-test3", &...
kekcment's user avatar
0 votes
0 answers
20 views

Confluent Kafka S3 source connector - not a valid AVRO format

I am trying to setup a Confluent kafka s3 source connector to read from S3 and add messages to Kafka topic. I am struggling with the formats - it asks for input and output format. What those mean? I ...
manpreet's user avatar
  • 636
0 votes
0 answers
39 views

Unable to initialize REST resources Error when trying to run kafka connect, connecting to Azura data storage sink on docker

I'm running docker with zookeeper, kafka, kafka-connect and azure-storage-sink. zookeeper and kafka run successfully but kafka-connect and azure-storage-sink containers exit with this error: [2023-08-...
Emc's user avatar
  • 11
0 votes
0 answers
48 views

Kafka/Snowflake Connector: "snowflake.schema.name schema does not exist"

I try to use the Snowflake Connector (snowflake-kafka-connector-2.0.0.jar ) in my Kafka (kafka_2.13-3.5.0). But when starting the connector, I get: "snowflake.schema.name schema does not exist&...
Wolfgang's user avatar
0 votes
0 answers
40 views

Debezium source connector database migrations breaking AVRO schema compatibility

Sorry for the stupid title. We're using Debezium source connector to stream changes from PostgreSQL to kafka. Along the way the connector is registering schemas to confluent schema-registry. Streaming ...
welcomeboredom's user avatar
0 votes
1 answer
86 views

Kafka connect Graceful stop of task failed

Context: using kafka connect 7.4.0 to read data from Postgres views and write to another Postgres tables. All works good, but whenever I stop the source connectors via the rest api, I always get: ...
Robson Hermes's user avatar
0 votes
0 answers
53 views

Migrate Kafka cluster to Azure Eventhubs

My objective is to migrate data from a kafka cluster to Azure eventhub . I tried MirrorMaker2 , it works but it requires namespace level admin privileges to work as MirrorMaker creates some internal ...
redeemed's user avatar
  • 471
0 votes
0 answers
25 views

Kafka connect is not consuming data continuously

As shown in the attached diagram Kafka Connect is consuming data only every 5 mins, any idea how to fix this and ensure continuous consumption.
Suraj's user avatar
  • 97
0 votes
0 answers
28 views

Kafka Connector - why is consumer creating a topic even though connector is not existing?

Im confused at this subject. One of my services has a consumer which listens on a topic. Im checking the topics in CMAK. If my consumer is up and running and if I delete the topic in CMAK it instantly ...
Furkan YIlmaZ's user avatar
0 votes
1 answer
41 views

Failed to deserialize data for topic "testtopic" to Avro for sink connector

Here is the sink connector properties "name": "sink-testtopic-crdb", "config": { "connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", &...
Sivapandian Jayakumar's user avatar
1 vote
2 answers
40 views

Is there any way to get the bad record which is causing the issue while sending it to the topic using source connector?

In sink connector(I am using Jdbc Sink Connector), if I use the following property I will get the bad or error causing record in the mentioned topic(i,e dead_topic). "errors.tolerance": &...
Muhamed Risvan M S's user avatar

1
2 3 4 5
75