3,867
questions
0
votes
0
answers
4
views
Single trace is not propagated from mongo-Kafka- sink-connector to mongoDb
I am using Mongo kafka sink connector to send the messages from kafka topic to mongodb.
I have integrate the openetelemetry agent with kafka connector with following environment variable
export ...
0
votes
0
answers
14
views
Kafka Connect sink connector filter for nested json
I have a sink connector to push data from a Kafka Topic to a Snowflake DB, my problem is that i need to add a transformation to filter out certain records based on a nested json.
A typical record ...
0
votes
0
answers
31
views
Snowflake kafka connector encrypted aes256 key fips 140-2 compliant?
Following the below url to set up Snowflake kafka connector
snowflake kafka connector
I then use the private key and pass phrase in the connector config and deploy to connect instance
I see the ...
0
votes
1
answer
13
views
Extracting Key and Appending It to the Message Value in Kafka Connect
I want to extract key and append it to the value by Kafka Connect. I read SMT and test some SMT but I can not do it.
I send this record value :
{"name":"ali"}
by this key :
Person
...
1
vote
0
answers
15
views
Logs not printed for Strimzi Kafkaconnect message transformer
I am using Strimzi kafkaconnect I have created Message Transformer but its logs are not getting printied in console how can i get those logs ?
following is log4j.properties file inside opt/kafka/...
1
vote
0
answers
30
views
kafka connector debezium stuck at snapshot of large data
I setup elasticsearch, kibana, mongodb, and kafka on the same linux server for development purposes. The server has 30GB Memory and enough disk space. I'm using a debezium connector and I'm trying ...
1
vote
0
answers
26
views
Data filtering in Debezium source connector (PostgreSQL)
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "<host>",
"database.port": "<port&...
1
vote
0
answers
18
views
I'm working on enabling change data capture from MySQL to Apache Kafka with Debezium but im getting an error in the sink connector
Error image
The error message is as follows:
[2024-09-03 06:57:22,110] ERROR [jdbc-connector|task-0] Failed to process record: 'org.apache.kafka.clients.consumer.ConsumerRecord org.apache.kafka....
0
votes
0
answers
34
views
About camel kafka source connector
I want to use the Camel HTTP Source Connector as the HTTP Source Connector among the Kafka Connectors.
As a result of REST Endpoint, json is as follows.
{
"list": [
{
...
0
votes
0
answers
13
views
problem with TimeBasePartitioner in s3 sink connector
I want to use TimeBasePartitioner and use RecordField for partitioning.
here is my configs:
"partition.duration.ms": "86400000",
"partitioner.class": "io.confluent....
1
vote
1
answer
27
views
Can we attach a init-container with strimzi kafka container?
Can we attach a init-container with strimzi kafka connect container? I want to copy some jars from another image (docker/my-sdk:9.1) to strimzi kafka connect container at runtime using init-container
...
0
votes
1
answer
19
views
ERROR Uncaught exception in REST call to /metrics (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
I am running a kafka connector in distributed mode, and explicitly disabled the jmx metrics exposure, still the /metrics endpoint is getting hit in every 45,12 seconds interval and returning 404 ...
0
votes
1
answer
40
views
How to receive tombstone messages in debezium connector?
Im using Kafka and Debezium Postgres Connector.
I wanted to extract only record value without other details, so I am using the "transforms.unwrap.type": "io.debezium.transforms....
2
votes
1
answer
24
views
How to configure Apache Kafka File Sink Connector to write JSON data as pipe-separated string and rotate files hourly?
I'm working with Apache Kafka and need to configure the File Sink Connector to achieve the following:
Convert JSON data to pipe-separated strings: The Kafka topic contains messages in JSON format, ...
0
votes
1
answer
27
views
Kafka Connect docker image not recognizing AWS environment variable credentials?
I'm currently learning Kafka Connect on my own by trying to set up some basic pipelines in Docker. The main issue is that it appears that when I try to use the connect REST API to spin up a new ...
1
vote
1
answer
31
views
How Kafka Connect Cluster is operated in production (number, distribution, etc.)
Hi I have 4 DBs and I need to connect all of them with Debezium Connector. I'm going to float them on AWS EKS and I'm also going to have S3 Sink Connectors, which I expect to be more than 10, divided ...
0
votes
0
answers
19
views
JDBC sink connector failed - Unknown Magic Byte
I use Kafka docker-compose stack. Confluent 7.7.0
and I want to use JDBC sink (confluentinc/kafka-connect-jdbc)
my source connector is debezium/debezium-connector-postgresql
and I faced with this ...
-1
votes
0
answers
24
views
ERROR finding table in UPPERCASE schema in Postgres using Kafka with Debezium connectors
I'm working with Kafka and Debezium connectors for CDC events in a PostgreSQL database, where the schema is in uppercase format like "SCHEMA". The Debezium source connector successfully ...
0
votes
0
answers
24
views
Debezium Kafka Connect / Mongo Connector silently failing, only spawning 1 worker, not creating topics, no errors
I am currently trying to deploy a 2.4.0.FINAL Debezium connector onto a Mongo 4.4 version instance.
However, my connector does not seem to work. I have metadata topics (ie, connect-offsets and connect-...
-1
votes
1
answer
62
views
Apache camel salesforce connector configuration debugging
I am trying to deploy the camel-salesforce-source connector.
I have downloaded the camel-salesforce-source-kafka-connector-4.4.2-package.tar.gz tar file from maven, extracted it to a dir and added the ...
1
vote
2
answers
59
views
Confluent Kafka: Data from Topic (single payload) to multiple tables
Does confluent kafka jdbc connector (sink) allow you to pull data from a topic and then define which tables on an oracle database to distribute that data to?.
i.e.
A single topic record will have a ...
0
votes
0
answers
26
views
How to use S3 kafka Connector Freely
I'm learning Kafka and creating a POC where I need to sink Kafka topics data to S3 and from S3 to Kafka(another cluster).
I see connectors are there from confluent but seems those are commercialized ...
0
votes
0
answers
21
views
How to update a record in a mysql table with Confluent JDBC sink connector only if the record I'm going to insert is greater than the current record
I would like to know if it was possible to update a record via Confluent's jdbc sink connector only if the value I am inserting is greater than or equal to the current one. This is because my KSQLDB ...
1
vote
1
answer
46
views
Access kafka cluster topic in confluent cloud
I'm trying to connect iceberg-kafka-connect connector to the kafka topic present in my confluent cloud. I'm using MSK connect to run this connector.
(I was able to access the confluent kafka topic ...
0
votes
0
answers
31
views
How to solve error on oracle-debezium llowing error: java.sql.SQLExcep ORA-00600: internal error code, arguments: [krvrdccs10]?
I'm trying to enable CDC (Debezium) in oracle 18c database ? I am getting the following error:
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor-java \n\tat java.base/...
0
votes
0
answers
20
views
Running a Kafka Connect worker without automatically creating topics
I need to run Kafka connect worker on a new port. The worker configuration is as follows:
bootstrap.servers=some-kafka-server.com:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value....
0
votes
1
answer
54
views
Kafka Connect JDBC sink date/time conversion to PostgreSQL timestamp with timezone
We have a Kafka topic with Cloud Events in JSON Schema format that we connected to Timescaledb/PostgreSQL with JDBC sink.
To convert a table to a hypertable we need a time column, but our date time ...
0
votes
0
answers
25
views
Confluent BigQuery sink connector is not creating nor updating tables properly when run in AWS MSK, but will work locally when running in Docker
I am running the Confluent BQ sink connector both in AWS MSK, and in Docker running Kafka Connect in distributed mode.
In distributed mode on my machine, the connector will run fine. It creates a new ...
0
votes
0
answers
33
views
org.apache.kafka.connect.errors.ConnectException: Value schema must be of type Struct
I recieve the data in my topic like this :
{"C_IST":"1",
"C_AG":"00",
"C_TP_RAPP":"1",
"N_RAPP":"3587",
"...
0
votes
0
answers
22
views
kafka-clients 3.4.0 Node X disconnected messages frequently
the log【INFO [Producer clientId=producer-1] Node X disconnected】.
springboot version is 2.7.18
kafka-clents version is 3.4.0
the producer all are working fine. is the info message harmful to the ...
0
votes
1
answer
65
views
Writing debezium CDC events from Kafka Connect to Iceberg
We want to ingest Debezium CDC events into Iceberg Tables on Azure Blob. We aim to do so using Kafka Connect Iceberg Sink Connector, and a Snowflake Catalog for Iceberg.
Questions that have surfaced -
...
0
votes
0
answers
15
views
Kafka Connect connector won't link up to a topic
In my work project I have some ksqlDB queries that rely on data provided by a topic that's supposed to be populated by a connector. Said connector isn't created through the ksqlDB CLI, but through the ...
0
votes
0
answers
80
views
Using the Confluent BigQuery sink Kafka connector, why are my messages/records not being written to new BQ tables, but existing tables are?
I am running Kafka Connect locally using Docker, but also in AWS MSK. Both are producing the same results.
My connector configuration is as follows:
{
"name": "connector-name",
&...
0
votes
0
answers
23
views
S3 Sink Confluent Kafka Connector JsonParser feature
I have json data in kafka, getting below stack trace
at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:331)
... 18 more Caused by: com.fasterxml.jackson.core....
-1
votes
1
answer
57
views
Strimzi Kafka Connect Cluster - Resource planning [closed]
I'm planing resources which will be dedicated to run Kafka Connect Clusters in Strimzi Distribution.
My workloads are pretty small, there will be probably only 1 task per connector needed.
I would go ...
1
vote
1
answer
146
views
Can debezium/Kafka connect ignore or resize too large messages?
We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL.
For 99% of our data this all works fine, but very occasionally debezium fails with:
[Worker-...
0
votes
1
answer
50
views
Getting Timeout Kafka exception org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA)
I am facing kafka timeout exception. Error is:
partition: B3-0 error: org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA), deadlineMs=1721138713548, tries=721498,...
0
votes
1
answer
42
views
How many kafka connect clusters are optimal?
I use Kafka with Strimzi operator and I am wondering if there is a optimal number of kafka connect clusters I should make. I have a medium sized infrastructure with around 50 different business logic ...
0
votes
1
answer
59
views
Minimal MirrorSourceConnector configurations to replicate data to another Kafka instance
I am trying to replicate topics from one Kafka cluster to another using Mirror Maker.
I am using docker compose to create the following resources:
networks:
local_kafka:
name: local_kafka
...
0
votes
0
answers
24
views
Add headers from kafka records to mongo field using sink connector
Im using a mongo sink connector to sink data from kafka topic to mongo collection, all i need is to add headers from these Kafka record to some field in the mongo record.
Its ok if I can map each ...
0
votes
1
answer
46
views
SQL connector to KSQLDB , how to do it in docker.yml?
I found myself struggling to get documentation updated on how to put SQL server DB to KSQLDB ?
I would like to know what would be the easiest way to get the connection toward Azure SQL Server and ...
0
votes
0
answers
102
views
Kafka Connect to Snowflake connection via JDBC error
I've been trying to send data from Kafka to Snowflake using the JDBC driver with Kafka Connect.
Some details about the environment:
Kafka is running in a Cloudera private cluster (Base 7.1.9).
The ...
0
votes
0
answers
15
views
how to prevent terminal stop when error occur during the kafka connection?
1) kafka code
import { Injectable } from "@nestjs/common";
import { v4 as uuidv4 } from 'uuid';
import { KafKaQueueNames } from "../dto/enum/kakaQueueNames";
import { Kafka, ...
0
votes
1
answer
104
views
MSK connector failed to find class in MSK cluster with KRaft mode
I have a Kafka 3.7.x cluster using ZooKeeper mode which works well. Now I am trying new Kafak in KRaft mode:
However, for this Kafka cluster in KRaft mode, the same connector failed to start.
Here ...
-1
votes
1
answer
63
views
Ansible EDA Kafka Connection Error - trying to create a connection to kafka within a rulebook but the rulebook activation fails
I have been trying to create a connection to kafka within a rulebook but when I run my rulebook activation I experience this error:
ansible_rulebook.rule_set_runner - INFO - Ruleset: Listen for events,...
0
votes
0
answers
44
views
Debezium stops after inserted or update data into the table when using decoderbufs
The initial synchronization worked as expected, but when I insert or update data into the table, the connection with postgres closes.The connector is still marked as active and running.This issue only ...
0
votes
0
answers
25
views
Kafka Connector use the same topic between debezium source and sink postgres
I have two kafka connector : debezium source and sink postgres. I would like to add a record on bdd kc-source, the record would be on db kc-sink, it is use the same topic.
kafka connector sink ...
2
votes
0
answers
84
views
Snowflake Kafka Connector private keys environmental variables unable to decrypt
I currently have my snowflake connector configuration set in a JSON file
{
"name": "connector-name",
"config": {
"tasks.max":"1",
"...
1
vote
0
answers
40
views
Kafka GCS sink connector flush size
I am using JDBC source connector to write the data into Kafka Topic for every 24 hours polling interval set and consuming the messages from Kafka topic using GCS sink connector. There is approximately ...
0
votes
0
answers
57
views
Error NOT_LEADER_OR_FOLLOWER in kafka connect-standalone
I run connect-stanalone.sh with configuration and after change db, Error "NOT_LEADER_OR_FOLLOWER" logged in console
bin/connect-standalone.sh config/connect-standalone.properties mongodb-...