Questions tagged [debezium]

Debezium is an open source distributed platform to stream database changes. Use this tag for questions related to using, configuring or troubleshooting Debezium.

Filter by
Sorted by
Tagged with
0 votes
1 answer
9 views

Debezium MSSQL "Producer failure Column not found in result set" after schema change and incremental snapshot

After adding a column to one of the source tables for an MSSQL Debezium connector, and then attempting an incremental snapshot on the table, the connector fails with the following error: [2023-03-27 ...
  • 3,761
0 votes
1 answer
15 views

Strimzi KafkaConnector not reading database credentials from secrets

I'm practicing running Debezium following the Official Documentation using Docker Desktop. The commands are pretty much just copy and paste from the page except that the output registry and plugin URL ...
-1 votes
0 answers
8 views

How to convert debezium messages into stateful rows

Is there a plugin or a library for redshift or python that aids in converting debezium WAL log messages into stateful rows in a database? I don't want to store the debezium messages directly in a ...
  • 10.7k
0 votes
0 answers
33 views

Debezium Outbox Pattern property transforms.outbox.table.expand.json.payload remove empty arrays

I'm implementing an outbox pattern using the debezium postgres connector, building up upon the official documentation: https://debezium.io/documentation/reference/stable/transformations/outbox-event-...
2 votes
0 answers
24 views

Couchbase Sink Kafka Connector Ignore Error and Continue

I created a sink connector from Kafka to Couchbase. It writes all events to Couchbase successfully until an event that has empty key came. It stuck at that point with failure on the task. Error in the ...
  • 421
0 votes
1 answer
36 views

Debezium Postgres confusion around snapshot & batch config

Could anyone explain what is the difference between those 4 variables and their co-dependencies when doing a snapshot with Postgres ? Is batch and queue size is being ignored during snapshot ? max....
0 votes
0 answers
20 views

Debezium connect retry (kafka degraded connection)

I have some occasional problems with some Debezium connections. Sometimes, people from another team at my company need to do some kind of maintenance on the source database, which I have a Debezium ...
0 votes
1 answer
58 views

Replicating two or more oracle tables in one flat data structure in Redis

I have two tables in oracle employee_info and employee_detail and want to replicate data of both the tables into single data structure(JSON- preferably) in Redis . I don't want to create Materialized ...
1 vote
1 answer
19 views

ksqlDB with Apicurio Registry

I’m trying to make ksqlDB work with Apicurio Registry (RedHat) and have had a few issues with compatibility with Debezium. So, Apicurio has 4 different APIs that can be used. The “official” and ...
0 votes
2 answers
45 views

How to stream mysql events to redis using debezium?

We are trying to use mysql as source db and redis as sink to build an ETL I have specified the sink as redis using following properties debezium.sink.type=redis debezium.sink.redis.batch.size=500 ...
0 votes
0 answers
30 views

syntax error at or near "GENERATED" with debezium/postgres container

I want to create app using debezium and kafka with dotnet core app. Kafka, postgres, zookeper and connector running on docker without problem. I tried to this coomand dotnet ef database update --...
  • 73
0 votes
0 answers
21 views

PyFlink Table API: Tumble window on joined debezium streams not producing output

I have two streams that have kafka as source and format debezium-json. The schema was excluded from the message. This is the definition of both streams: CREATE TABLE transactions ( `account_id`...
0 votes
1 answer
111 views

Failed to resolve 'kafka:9092': nodename nor servname provided, or not known [duplicate]

I want to run kafka and debezium in docker. I write to try docker-compose file but i'm getting this error 'Failed to resolve 'kafka:9092': nodename nor servname provided, or not known '. In my code ...
  • 73
3 votes
1 answer
83 views

AvroConverter fails to serialize Nested Structures using latest schema version

I'm trying to setup a debezium connector to capture the changes on a collection and stream those changes to a kafka topic. Everything works great (inserts, updates, deletes/tombstones) until I ...
  • 63
0 votes
0 answers
49 views

Debezium connector for SQL server

I have a confluent platform 7.1 installed by ansible I'm trying to run the Debezium 2.1 connector for SQL server using SASL_SSL authentication, when I ran the connector without authentication it works ...
0 votes
1 answer
45 views

JDBC Sink Connector ConnectException - Table "schema"."table" is missing and auto-creation is disabled for oracle

I am trying to sink data from kafka topic "kafka2.INVENTORY.PRODUCT" to table "product_target" under schema name "tinventory" with Oracle DB. This sink I am trying with ...
  • 69
1 vote
1 answer
43 views

Debezium is failing after updating connector that use confluence schema registry

I have a kubernetes cluster that stream my database events where I setup debezium and a confluence schema registry. They connect directly to a kafka cluster inside confluent platform. Every time I ...
0 votes
2 answers
56 views

Can there be 2 Debezium connector running in one Kafka cluster?

From Debezium official page, there's this picture showing that multiple Debezium connector can connect to the same Kafka. So I have 2 databases, 2 Debeziums, 1 Kafka running in docker-compose, but it ...
0 votes
0 answers
15 views

debezium mysql 2.1.2 data latency is stable at 1 minute

When I use debezium mysql 2.1.2, by observing the message timestamp in kafka and comparing it with the writing time of the data in the database, I found that the change will be sent to kafka after 1 ...
  • 1
0 votes
0 answers
32 views

Add connector json configuration to Kafka Connect REST API via gradle

I want to use Debezium and Kafka Connect to implementing transaction outbox pattern. I also use Postgres. According to the official documentation, I have to call a REST endpoint to register our ...
  • 821
1 vote
1 answer
137 views

Debezium MySQL connector error: Encountered change event for table whose schema isn't known to this connector

I've created a Debezium connector to listen binlog event from Mysql by this config: { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "topic.creation....
  • 676
1 vote
1 answer
47 views

Synchning Postgres & Elastic Search with Kafka Connect ( Debezium )

I have set up using docker a postgres image, aswell as a elastic search one. What i'm trying to achieve is that i have a Vehicle entity ( on a microservice with spring data jpa) , as well as Vehicle ...
  • 55
0 votes
1 answer
47 views

How can I get specific header from response body in Kafka Connector

There is a response which is belong apache kafka-connect, I want to get just "ID" from that body. Is there any way in order to eliminate that body. {"schema":{"type":&...
0 votes
1 answer
22 views

Can 1 debezium container connect to multiple postgres databases?

Says that I have 3 different Postgres database hosting on a different cloud providers. So I wonder if I could use 1 Debezium container to connect to these 3 databases to send datas to Kafka ? If yes, ...
3 votes
2 answers
332 views

AWS MSK Connect w/ MSSQL Debezium connector fails with disconnect

I am trying to setup a mssql debezium connector with AWS MSK Connect but keep getting the following error messages: Connector error log: [Worker-0a949760f6b805d4f] [2023-02-15 19:57:56,122] WARN [src-...
0 votes
0 answers
17 views

CDC replication not working after few days of inactivity

I am struggling with a issue where Debezium is unable to continue CDC replication after a few days of inactivity on source DB (SQL Server Always ON). Debezium Version: 1.9.5 The Debezium is reading ...
  • 57
0 votes
2 answers
170 views

Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are

Setting up Data Streaming Pipeline using Kafka, MySQL and Debezium. I'm this version of Kafka - 3.4.0, MySQL - 8, Debezium - 2.2.1, Java - 11. AIM : I want to capture all CDC from MySQL and stream the ...
0 votes
0 answers
14 views

Unable to start debezium connector in distributed mode

Trying to deploy debezium using kafka connect distributed mode is causing issues . The connect worker shut down with not a clear exception. The group-id decleration along with the topic.regex is not ...
0 votes
0 answers
77 views

Data Streaming - Debezium MySQL Connector Configuration to stream CDC to kafka Topics [duplicate]

I'm setting up data streaming pipeline with Kafka, Debezium MySQL Connector and MySQL. When the data is inserted into MySQL, the Debezium MySQL Connector(CDC) need to push the data to Kafka topic and ...
0 votes
0 answers
12 views

How to create a Postgres -> Kafka connector with custom configuration?

heroku data:connectors:create command only has "Tables to include" and "Columns to exclude" options. But I need debezium connector configured with transforms, converters, and ...
  • 170
0 votes
1 answer
39 views

NPE when encrypting fields for SQLServer databases using debezium

I am using debezium to do a CDC from SQLServer to kafka, and as per the business needs, some of the columns must be encrypted. For the environment POV, I have 2 kafka-connect instances running on K8S, ...
  • 1,196
1 vote
0 answers
32 views

In Debezium op type is coming as null

I have a Debezium connector upon a MySql DB table. I am adding the op type to the payload this way: "transforms.unwrap.add.fields": "op", I am not sure if it is relevant, but I am ...
  • 316
0 votes
0 answers
17 views

Debezium MySQL: Add source database connection/user to message payload

Can the Debezium MySQL connector be configured so messages it emits contain metadata that can be used to infer which MySQL user performed the change? My use case is a CDC for a web application with a ...
  • 454
0 votes
1 answer
22 views

Construct own message

I have user table, which is obviously does not outbox table. Table consist uuid and email field. I need to construct message and publish into Kafka based on this two field with next format: { "...
0 votes
0 answers
45 views

Debezium mongodb connector delete event doesn’t give the before event in payload

adding a question to this thread.. I am using the debezium connetor and everything is going smooth except with few things. The delete operation doesn’t give any after or patch or before event and it ...
  • 7
0 votes
0 answers
31 views

Debezium transformation drop event filter not working as expected

I am trying to filter the CDC event based on payload values using transformation. Below are jars and config I am using. But I don't see events getting published on kafka topic with this config(even ...
  • 69
0 votes
0 answers
37 views

debezium date/time field value out of range: 0000-12-30T00:00:00Z

We use debezium to sync data to In the source table we have column timestamptz start_at, when the value is zero start_at='0001-01-01 00:00:00.000000 +00:00', but when we check the data in kafka, it's ...
  • 1
0 votes
1 answer
35 views

how to distribute messages to all partitions in topic defined by `offset.storage.topic` in kafka connect

I have deployed debezium using the docker image pulled from docker pull debezium/connect In the documentation provided at https://hub.docker.com/r/debezium/connect the description for one of the ...
0 votes
1 answer
88 views

Kafka JDBC sink connector trying to update destination table on delete and fails

I am using Debezium source connector (postgres) to track database changes to kafka and I am using kafka jdbc sink connector to transfer the data to another postgres server. Here insert and update are ...
1 vote
1 answer
180 views

Kafka connect -Debezium- Oracle JDBC Sink connector error consuming debezium(oracle) CDC event

I have below images running in docker. quay.io/debezium/connect:2.0 quay.io/debezium/kafka:2.0 quay.io/debezium/zookeeper:2.0 container-registry.oracle.com/database/enterprise:latest debezium-source-...
  • 69
1 vote
1 answer
109 views

Debezium: Produce messages only upon changes to columns in column.include.list

I am running Debezium with column.include.list configured to a subset of columns on each of the observed tables of the source MySQL database. Changes to records on the source tables are being ...
  • 454
0 votes
0 answers
95 views

Debezium misssing some events

During the process of data discrepancy, I realised some events are missed and not written to kafka during CDC. These are the steps I did: 1- Setup two different debezium connector each fetch data ...
  • 335
0 votes
0 answers
29 views

Get tenant_id from a delete CDC operation in Debezium

I am trying to develop a SpringBoot application to track CDC changes of a Postgres database. In DebeziumConfig I have private io.debezium.config.Configuration config() { return io.debezium....
  • 181
0 votes
1 answer
92 views

Debezium is not working with redis streams and postgresql

I'm having an issue with running debezium with redis and postgresql. My docker compose is: version: "3.3" services: redis-stack: image: redis/redis-stack:7.0.6-RC4 restart: unless-...
  • 549
0 votes
0 answers
75 views

Kafka Connect JdbcSinkConnector - error on INSERT (invalid byte sequence)

We have data replication solution using Kafka Connect. Data is being read using Debezium SqlServerConnector into multiple topics, and then written to PostgreSQL using JdbcSinkConnector. Each topic/...
0 votes
1 answer
43 views

Errors with BigQuery Sink Connector Configuration

I am trying to ingest data from MySQL to BigQuery. I am using Debezium components running on Docker for this purpose. Anytime I try to deploy the BigQuery sink connector to Kafka connect, I am getting ...
  • 23
0 votes
1 answer
86 views

Confluent connector tasks have failed

Started the docker containers (cp-kafka-connect-base:7.0.1) and installed the self-managed connector (debezium-connector-mysql:latest). Docker containers works good. When I am trying to configure the ...
0 votes
0 answers
69 views

Debezium fails parsing a RENAME query

I deployed a Debezium MySQL connector to capture changes for table A (database has many more tables). I see the following exception: [Worker-0b2620d26c36f7ef3] io.debezium.text.ParsingException: ...
  • 5,473
0 votes
0 answers
62 views

Sink Connector auto create tables with proper data type

I have the debezium source connector for Postgresql with the value convertor as Avro and it uses the schema registry. Source DDL: Table ...
  • 2,520
0 votes
0 answers
44 views

Updating Debezium MySQL connector to resume with database moved to RDS

I have moved my MySQL database to RDS. I would like to update the connector that publishes messages from an outbox table to a specific Kafka topic to point to the RDS instance. What would be the steps ...

1
2 3 4 5
20