Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Single trace is not propagated from mongo-Kafka- sink-connector to mongoDb

I am using Mongo kafka sink connector to send the messages from kafka topic to mongodb. I have integrate the openetelemetry agent with kafka connector with following environment variable export ...
jiten gawade's user avatar
0 votes
0 answers
14 views

Kafka Connect sink connector filter for nested json

I have a sink connector to push data from a Kafka Topic to a Snowflake DB, my problem is that i need to add a transformation to filter out certain records based on a nested json. A typical record ...
matthewr's user avatar
  • 312
0 votes
0 answers
31 views

Snowflake kafka connector encrypted aes256 key fips 140-2 compliant?

Following the below url to set up Snowflake kafka connector snowflake kafka connector I then use the private key and pass phrase in the connector config and deploy to connect instance I see the ...
md1980's user avatar
  • 329
0 votes
1 answer
13 views

Extracting Key and Appending It to the Message Value in Kafka Connect

I want to extract key and append it to the value by Kafka Connect. I read SMT and test some SMT but I can not do it. I send this record value : {"name":"ali"} by this key : Person ...
Alihossein shahabi's user avatar
1 vote
0 answers
15 views

Logs not printed for Strimzi Kafkaconnect message transformer

I am using Strimzi kafkaconnect I have created Message Transformer but its logs are not getting printied in console how can i get those logs ? following is log4j.properties file inside opt/kafka/...
Gurpreet Singh Sran's user avatar
1 vote
0 answers
30 views

kafka connector debezium stuck at snapshot of large data

I setup elasticsearch, kibana, mongodb, and kafka on the same linux server for development purposes. The server has 30GB Memory and enough disk space. I'm using a debezium connector and I'm trying ...
Osama's user avatar
  • 476
1 vote
0 answers
26 views

Data filtering in Debezium source connector (PostgreSQL)

{ "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "<host>", "database.port": "<port&...
ESSAKI NATARAJAN's user avatar
1 vote
0 answers
18 views

I'm working on enabling change data capture from MySQL to Apache Kafka with Debezium but im getting an error in the sink connector

Error image The error message is as follows: [2024-09-03 06:57:22,110] ERROR [jdbc-connector|task-0] Failed to process record: 'org.apache.kafka.clients.consumer.ConsumerRecord org.apache.kafka....
Hariharan Hari's user avatar
0 votes
0 answers
34 views

About camel kafka source connector

I want to use the Camel HTTP Source Connector as the HTTP Source Connector among the Kafka Connectors. As a result of REST Endpoint, json is as follows. { "list": [ { ...
ceounii lee's user avatar
0 votes
0 answers
13 views

problem with TimeBasePartitioner in s3 sink connector

I want to use TimeBasePartitioner and use RecordField for partitioning. here is my configs: "partition.duration.ms": "86400000", "partitioner.class": "io.confluent....
Nestor's user avatar
  • 73
1 vote
1 answer
27 views

Can we attach a init-container with strimzi kafka container?

Can we attach a init-container with strimzi kafka connect container? I want to copy some jars from another image (docker/my-sdk:9.1) to strimzi kafka connect container at runtime using init-container ...
Roobal Jindal's user avatar
0 votes
1 answer
19 views

ERROR Uncaught exception in REST call to /metrics (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)

I am running a kafka connector in distributed mode, and explicitly disabled the jmx metrics exposure, still the /metrics endpoint is getting hit in every 45,12 seconds interval and returning 404 ...
Gopal's user avatar
  • 3
0 votes
1 answer
40 views

How to receive tombstone messages in debezium connector?

Im using Kafka and Debezium Postgres Connector. I wanted to extract only record value without other details, so I am using the "transforms.unwrap.type": "io.debezium.transforms....
Sara M.'s user avatar
  • 303
2 votes
1 answer
24 views

How to configure Apache Kafka File Sink Connector to write JSON data as pipe-separated string and rotate files hourly?

I'm working with Apache Kafka and need to configure the File Sink Connector to achieve the following: Convert JSON data to pipe-separated strings: The Kafka topic contains messages in JSON format, ...
Vishnu's user avatar
  • 21
0 votes
1 answer
27 views

Kafka Connect docker image not recognizing AWS environment variable credentials?

I'm currently learning Kafka Connect on my own by trying to set up some basic pipelines in Docker. The main issue is that it appears that when I try to use the connect REST API to spin up a new ...
user3491700's user avatar
1 vote
1 answer
31 views

How Kafka Connect Cluster is operated in production (number, distribution, etc.)

Hi I have 4 DBs and I need to connect all of them with Debezium Connector. I'm going to float them on AWS EKS and I'm also going to have S3 Sink Connectors, which I expect to be more than 10, divided ...
Suhyun's user avatar
  • 13
0 votes
0 answers
19 views

JDBC sink connector failed - Unknown Magic Byte

I use Kafka docker-compose stack. Confluent 7.7.0 and I want to use JDBC sink (confluentinc/kafka-connect-jdbc) my source connector is debezium/debezium-connector-postgresql and I faced with this ...
mahradbt's user avatar
  • 400
-1 votes
0 answers
24 views

ERROR finding table in UPPERCASE schema in Postgres using Kafka with Debezium connectors

I'm working with Kafka and Debezium connectors for CDC events in a PostgreSQL database, where the schema is in uppercase format like "SCHEMA". The Debezium source connector successfully ...
ESSAKI NATARAJAN's user avatar
0 votes
0 answers
24 views

Debezium Kafka Connect / Mongo Connector silently failing, only spawning 1 worker, not creating topics, no errors

I am currently trying to deploy a 2.4.0.FINAL Debezium connector onto a Mongo 4.4 version instance. However, my connector does not seem to work. I have metadata topics (ie, connect-offsets and connect-...
antonig's user avatar
  • 163
-1 votes
1 answer
62 views

Apache camel salesforce connector configuration debugging

I am trying to deploy the camel-salesforce-source connector. I have downloaded the camel-salesforce-source-kafka-connector-4.4.2-package.tar.gz tar file from maven, extracted it to a dir and added the ...
Anup's user avatar
  • 945
1 vote
2 answers
59 views

Confluent Kafka: Data from Topic (single payload) to multiple tables

Does confluent kafka jdbc connector (sink) allow you to pull data from a topic and then define which tables on an oracle database to distribute that data to?. i.e. A single topic record will have a ...
David Garcia's user avatar
  • 2,686
0 votes
0 answers
26 views

How to use S3 kafka Connector Freely

I'm learning Kafka and creating a POC where I need to sink Kafka topics data to S3 and from S3 to Kafka(another cluster). I see connectors are there from confluent but seems those are commercialized ...
Lalit Shukla's user avatar
0 votes
0 answers
21 views

How to update a record in a mysql table with Confluent JDBC sink connector only if the record I'm going to insert is greater than the current record

I would like to know if it was possible to update a record via Confluent's jdbc sink connector only if the value I am inserting is greater than or equal to the current one. This is because my KSQLDB ...
MrRobot's user avatar
1 vote
1 answer
46 views

Access kafka cluster topic in confluent cloud

I'm trying to connect iceberg-kafka-connect connector to the kafka topic present in my confluent cloud. I'm using MSK connect to run this connector. (I was able to access the confluent kafka topic ...
Malina Dale's user avatar
0 votes
0 answers
31 views

How to solve error on oracle-debezium llowing error: java.sql.SQLExcep ORA-00600: internal error code, arguments: [krvrdccs10]?

I'm trying to enable CDC (Debezium) in oracle 18c database ? I am getting the following error: java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor-java \n\tat java.base/...
Evandro Terra's user avatar
0 votes
0 answers
20 views

Running a Kafka Connect worker without automatically creating topics

I need to run Kafka connect worker on a new port. The worker configuration is as follows: bootstrap.servers=some-kafka-server.com:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value....
langepass88's user avatar
0 votes
1 answer
54 views

Kafka Connect JDBC sink date/time conversion to PostgreSQL timestamp with timezone

We have a Kafka topic with Cloud Events in JSON Schema format that we connected to Timescaledb/PostgreSQL with JDBC sink. To convert a table to a hypertable we need a time column, but our date time ...
Piotr Kozlowski's user avatar
0 votes
0 answers
25 views

Confluent BigQuery sink connector is not creating nor updating tables properly when run in AWS MSK, but will work locally when running in Docker

I am running the Confluent BQ sink connector both in AWS MSK, and in Docker running Kafka Connect in distributed mode. In distributed mode on my machine, the connector will run fine. It creates a new ...
Sultan of Swing's user avatar
0 votes
0 answers
33 views

org.apache.kafka.connect.errors.ConnectException: Value schema must be of type Struct

I recieve the data in my topic like this : {"C_IST":"1", "C_AG":"00", "C_TP_RAPP":"1", "N_RAPP":"3587", "...
Alaa Saeed's user avatar
0 votes
0 answers
22 views

kafka-clients 3.4.0 Node X disconnected messages frequently

the log【INFO [Producer clientId=producer-1] Node X disconnected】. springboot version is 2.7.18 kafka-clents version is 3.4.0 the producer all are working fine. is the info message harmful to the ...
star's user avatar
  • 1
0 votes
1 answer
65 views

Writing debezium CDC events from Kafka Connect to Iceberg

We want to ingest Debezium CDC events into Iceberg Tables on Azure Blob. We aim to do so using Kafka Connect Iceberg Sink Connector, and a Snowflake Catalog for Iceberg. Questions that have surfaced - ...
gt1's user avatar
  • 3
0 votes
0 answers
15 views

Kafka Connect connector won't link up to a topic

In my work project I have some ksqlDB queries that rely on data provided by a topic that's supposed to be populated by a connector. Said connector isn't created through the ksqlDB CLI, but through the ...
Emanuele Righetti's user avatar
0 votes
0 answers
80 views

Using the Confluent BigQuery sink Kafka connector, why are my messages/records not being written to new BQ tables, but existing tables are?

I am running Kafka Connect locally using Docker, but also in AWS MSK. Both are producing the same results. My connector configuration is as follows: { "name": "connector-name", &...
Sultan of Swing's user avatar
0 votes
0 answers
23 views

S3 Sink Confluent Kafka Connector JsonParser feature

I have json data in kafka, getting below stack trace at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:331) ... 18 more Caused by: com.fasterxml.jackson.core....
gaurav miglani's user avatar
-1 votes
1 answer
57 views

Strimzi Kafka Connect Cluster - Resource planning [closed]

I'm planing resources which will be dedicated to run Kafka Connect Clusters in Strimzi Distribution. My workloads are pretty small, there will be probably only 1 task per connector needed. I would go ...
repcak's user avatar
  • 144
1 vote
1 answer
146 views

Can debezium/Kafka connect ignore or resize too large messages?

We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL. For 99% of our data this all works fine, but very occasionally debezium fails with: [Worker-...
Austen's user avatar
  • 306
0 votes
1 answer
50 views

Getting Timeout Kafka exception org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA)

I am facing kafka timeout exception. Error is: partition: B3-0 error: org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA), deadlineMs=1721138713548, tries=721498,...
skd's user avatar
  • 165
0 votes
1 answer
42 views

How many kafka connect clusters are optimal?

I use Kafka with Strimzi operator and I am wondering if there is a optimal number of kafka connect clusters I should make. I have a medium sized infrastructure with around 50 different business logic ...
camel38's user avatar
  • 91
0 votes
1 answer
59 views

Minimal MirrorSourceConnector configurations to replicate data to another Kafka instance

I am trying to replicate topics from one Kafka cluster to another using Mirror Maker. I am using docker compose to create the following resources: networks: local_kafka: name: local_kafka ...
J. Snow's user avatar
  • 306
0 votes
0 answers
24 views

Add headers from kafka records to mongo field using sink connector

Im using a mongo sink connector to sink data from kafka topic to mongo collection, all i need is to add headers from these Kafka record to some field in the mongo record. Its ok if I can map each ...
Akshay Som's user avatar
0 votes
1 answer
46 views

SQL connector to KSQLDB , how to do it in docker.yml?

I found myself struggling to get documentation updated on how to put SQL server DB to KSQLDB ? I would like to know what would be the easiest way to get the connection toward Azure SQL Server and ...
HPAmaris's user avatar
0 votes
0 answers
102 views

Kafka Connect to Snowflake connection via JDBC error

I've been trying to send data from Kafka to Snowflake using the JDBC driver with Kafka Connect. Some details about the environment: Kafka is running in a Cloudera private cluster (Base 7.1.9). The ...
alex's user avatar
  • 1
0 votes
0 answers
15 views

how to prevent terminal stop when error occur during the kafka connection?

1) kafka code import { Injectable } from "@nestjs/common"; import { v4 as uuidv4 } from 'uuid'; import { KafKaQueueNames } from "../dto/enum/kakaQueueNames"; import { Kafka, ...
buvanesh's user avatar
0 votes
1 answer
104 views

MSK connector failed to find class in MSK cluster with KRaft mode

I have a Kafka 3.7.x cluster using ZooKeeper mode which works well. Now I am trying new Kafak in KRaft mode: However, for this Kafka cluster in KRaft mode, the same connector failed to start. Here ...
Hongbo Miao's user avatar
  • 49.1k
-1 votes
1 answer
63 views

Ansible EDA Kafka Connection Error - trying to create a connection to kafka within a rulebook but the rulebook activation fails

I have been trying to create a connection to kafka within a rulebook but when I run my rulebook activation I experience this error: ansible_rulebook.rule_set_runner - INFO - Ruleset: Listen for events,...
user26324472's user avatar
0 votes
0 answers
44 views

Debezium stops after inserted or update data into the table when using decoderbufs

The initial synchronization worked as expected, but when I insert or update data into the table, the connection with postgres closes.The connector is still marked as active and running.This issue only ...
lingfeng qian's user avatar
0 votes
0 answers
25 views

Kafka Connector use the same topic between debezium source and sink postgres

I have two kafka connector : debezium source and sink postgres. I would like to add a record on bdd kc-source, the record would be on db kc-sink, it is use the same topic. kafka connector sink ...
locklockls's user avatar
2 votes
0 answers
84 views

Snowflake Kafka Connector private keys environmental variables unable to decrypt

I currently have my snowflake connector configuration set in a JSON file { "name": "connector-name", "config": { "tasks.max":"1", "...
engicode123's user avatar
1 vote
0 answers
40 views

Kafka GCS sink connector flush size

I am using JDBC source connector to write the data into Kafka Topic for every 24 hours polling interval set and consuming the messages from Kafka topic using GCS sink connector. There is approximately ...
Adhish's user avatar
  • 87
0 votes
0 answers
57 views

Error NOT_LEADER_OR_FOLLOWER in kafka connect-standalone

I run connect-stanalone.sh with configuration and after change db, Error "NOT_LEADER_OR_FOLLOWER" logged in console bin/connect-standalone.sh config/connect-standalone.properties mongodb-...
md.119's user avatar
  • 21

1
2 3 4 5
78