Questions tagged [debezium]
Debezium is an open source distributed platform to stream database changes. Use this tag for questions related to using, configuring or troubleshooting Debezium.
977
questions
0
votes
1
answer
9
views
Debezium MSSQL "Producer failure Column not found in result set" after schema change and incremental snapshot
After adding a column to one of the source tables for an MSSQL Debezium connector, and then attempting an incremental snapshot on the table, the connector fails with the following error:
[2023-03-27 ...
0
votes
1
answer
15
views
Strimzi KafkaConnector not reading database credentials from secrets
I'm practicing running Debezium following the Official Documentation using Docker Desktop.
The commands are pretty much just copy and paste from the page except that the output registry and plugin URL ...
-1
votes
0
answers
8
views
How to convert debezium messages into stateful rows
Is there a plugin or a library for redshift or python that aids in converting debezium WAL log messages into stateful rows in a database? I don't want to store the debezium messages directly in a ...
0
votes
0
answers
33
views
Debezium Outbox Pattern property transforms.outbox.table.expand.json.payload remove empty arrays
I'm implementing an outbox pattern using the debezium postgres connector, building up upon the official documentation: https://debezium.io/documentation/reference/stable/transformations/outbox-event-...
2
votes
0
answers
24
views
Couchbase Sink Kafka Connector Ignore Error and Continue
I created a sink connector from Kafka to Couchbase.
It writes all events to Couchbase successfully until an event that has empty key came.
It stuck at that point with failure on the task.
Error in the ...
0
votes
1
answer
36
views
Debezium Postgres confusion around snapshot & batch config
Could anyone explain what is the difference between those 4 variables and their co-dependencies when doing a snapshot with Postgres ? Is batch and queue size is being ignored during snapshot ?
max....
0
votes
0
answers
20
views
Debezium connect retry (kafka degraded connection)
I have some occasional problems with some Debezium connections.
Sometimes, people from another team at my company need to do some kind of maintenance on the source database, which I have a Debezium ...
0
votes
1
answer
58
views
Replicating two or more oracle tables in one flat data structure in Redis
I have two tables in oracle employee_info and employee_detail and want to replicate data of both the tables into single data structure(JSON- preferably) in Redis .
I don't want to create Materialized ...
1
vote
1
answer
19
views
ksqlDB with Apicurio Registry
I’m trying to make ksqlDB work with Apicurio Registry (RedHat) and have had a few issues with compatibility with Debezium.
So, Apicurio has 4 different APIs that can be used. The “official” and ...
0
votes
2
answers
45
views
How to stream mysql events to redis using debezium?
We are trying to use mysql as source db and redis as sink to build an ETL
I have specified the sink as redis using following properties
debezium.sink.type=redis
debezium.sink.redis.batch.size=500
...
0
votes
0
answers
30
views
syntax error at or near "GENERATED" with debezium/postgres container
I want to create app using debezium and kafka with dotnet core app. Kafka, postgres, zookeper and connector running on docker without problem. I tried to this coomand dotnet ef database update --...
0
votes
0
answers
21
views
PyFlink Table API: Tumble window on joined debezium streams not producing output
I have two streams that have kafka as source and format debezium-json. The schema was excluded from the message.
This is the definition of both streams:
CREATE TABLE transactions (
`account_id`...
0
votes
1
answer
111
views
Failed to resolve 'kafka:9092': nodename nor servname provided, or not known [duplicate]
I want to run kafka and debezium in docker. I write to try docker-compose file but i'm getting this error 'Failed to resolve 'kafka:9092': nodename nor servname provided, or not known '. In my code ...
3
votes
1
answer
83
views
AvroConverter fails to serialize Nested Structures using latest schema version
I'm trying to setup a debezium connector to capture the changes on a collection and stream those changes to a kafka topic.
Everything works great (inserts, updates, deletes/tombstones) until I ...
0
votes
0
answers
49
views
Debezium connector for SQL server
I have a confluent platform 7.1 installed by ansible I'm trying to run the Debezium 2.1 connector for SQL server using SASL_SSL authentication, when I ran the connector without authentication it works ...
0
votes
1
answer
45
views
JDBC Sink Connector ConnectException - Table "schema"."table" is missing and auto-creation is disabled for oracle
I am trying to sink data from kafka topic "kafka2.INVENTORY.PRODUCT" to table "product_target" under schema name "tinventory" with Oracle DB. This sink I am trying with ...
1
vote
1
answer
43
views
Debezium is failing after updating connector that use confluence schema registry
I have a kubernetes cluster that stream my database events where I setup debezium and a confluence schema registry. They connect directly to a kafka cluster inside confluent platform.
Every time I ...
0
votes
2
answers
56
views
Can there be 2 Debezium connector running in one Kafka cluster?
From Debezium official page, there's this picture showing that multiple Debezium connector can connect to the same Kafka.
So I have 2 databases, 2 Debeziums, 1 Kafka running in docker-compose, but it ...
0
votes
0
answers
15
views
debezium mysql 2.1.2 data latency is stable at 1 minute
When I use debezium mysql 2.1.2, by observing the message timestamp in kafka and comparing it with the writing time of the data in the database, I found that the change will be sent to kafka after 1 ...
0
votes
0
answers
32
views
Add connector json configuration to Kafka Connect REST API via gradle
I want to use Debezium and Kafka Connect to implementing transaction outbox pattern. I also use Postgres.
According to the official documentation, I have to call a REST endpoint to register our ...
1
vote
1
answer
137
views
Debezium MySQL connector error: Encountered change event for table whose schema isn't known to this connector
I've created a Debezium connector to listen binlog event from Mysql by this config:
{
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"topic.creation....
1
vote
1
answer
47
views
Synchning Postgres & Elastic Search with Kafka Connect ( Debezium )
I have set up using docker a postgres image, aswell as a elastic search one.
What i'm trying to achieve is that i have a Vehicle entity ( on a microservice with spring data jpa) , as well as Vehicle ...
0
votes
1
answer
47
views
How can I get specific header from response body in Kafka Connector
There is a response which is belong apache kafka-connect, I want to get just "ID" from that body. Is there any way in order to eliminate that body.
{"schema":{"type":&...
0
votes
1
answer
22
views
Can 1 debezium container connect to multiple postgres databases?
Says that I have 3 different Postgres database hosting on a different cloud providers.
So I wonder if I could use 1 Debezium container to connect to these 3 databases to send datas to Kafka ?
If yes, ...
3
votes
2
answers
332
views
AWS MSK Connect w/ MSSQL Debezium connector fails with disconnect
I am trying to setup a mssql debezium connector with AWS MSK Connect but keep getting the following error messages:
Connector error log:
[Worker-0a949760f6b805d4f] [2023-02-15 19:57:56,122] WARN [src-...
0
votes
0
answers
17
views
CDC replication not working after few days of inactivity
I am struggling with a issue where Debezium is unable to continue CDC replication after a few days of inactivity on source DB (SQL Server Always ON). Debezium Version: 1.9.5
The Debezium is reading ...
0
votes
2
answers
170
views
Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are
Setting up Data Streaming Pipeline using Kafka, MySQL and Debezium. I'm this version of Kafka - 3.4.0, MySQL - 8, Debezium - 2.2.1, Java - 11. AIM : I want to capture all CDC from MySQL and stream the ...
0
votes
0
answers
14
views
Unable to start debezium connector in distributed mode
Trying to deploy debezium using kafka connect distributed mode is causing issues .
The connect worker shut down with not a clear exception.
The group-id decleration along with the topic.regex is not ...
0
votes
0
answers
77
views
Data Streaming - Debezium MySQL Connector Configuration to stream CDC to kafka Topics [duplicate]
I'm setting up data streaming pipeline with Kafka, Debezium MySQL Connector and MySQL. When the data is inserted into MySQL, the Debezium MySQL Connector(CDC) need to push the data to Kafka topic and ...
0
votes
0
answers
12
views
How to create a Postgres -> Kafka connector with custom configuration?
heroku data:connectors:create command only has "Tables to include" and "Columns to exclude" options. But I need debezium connector configured with transforms, converters, and ...
0
votes
1
answer
39
views
NPE when encrypting fields for SQLServer databases using debezium
I am using debezium to do a CDC from SQLServer to kafka, and as per the business needs, some of the columns must be encrypted.
For the environment POV, I have 2 kafka-connect instances running on K8S, ...
1
vote
0
answers
32
views
In Debezium op type is coming as null
I have a Debezium connector upon a MySql DB table. I am adding the op type to the payload this way:
"transforms.unwrap.add.fields": "op",
I am not sure if it is relevant, but I am ...
0
votes
0
answers
17
views
Debezium MySQL: Add source database connection/user to message payload
Can the Debezium MySQL connector be configured so messages it emits contain metadata that can be used to infer which MySQL user performed the change?
My use case is a CDC for a web application with a ...
0
votes
1
answer
22
views
Construct own message
I have user table, which is obviously does not outbox table. Table consist uuid and email field. I need to construct message and publish into Kafka based on this two field with next format:
{
"...
0
votes
0
answers
45
views
Debezium mongodb connector delete event doesn’t give the before event in payload
adding a question to this thread..
I am using the debezium connetor and everything is going smooth except with few things. The delete operation doesn’t give any after or patch or before event and it ...
0
votes
0
answers
31
views
Debezium transformation drop event filter not working as expected
I am trying to filter the CDC event based on payload values using transformation. Below are jars and config I am using. But I don't see events getting published on kafka topic with this config(even ...
0
votes
0
answers
37
views
debezium date/time field value out of range: 0000-12-30T00:00:00Z
We use debezium to sync data to
In the source table we have column timestamptz start_at, when the value is zero start_at='0001-01-01 00:00:00.000000 +00:00', but when we check the data in kafka, it's ...
0
votes
1
answer
35
views
how to distribute messages to all partitions in topic defined by `offset.storage.topic` in kafka connect
I have deployed debezium using the docker image pulled from docker pull debezium/connect
In the documentation provided at https://hub.docker.com/r/debezium/connect the description for one of the ...
0
votes
1
answer
88
views
Kafka JDBC sink connector trying to update destination table on delete and fails
I am using Debezium source connector (postgres) to track database changes to kafka and I am using kafka jdbc sink connector to transfer the data to another postgres server. Here insert and update are ...
1
vote
1
answer
180
views
Kafka connect -Debezium- Oracle JDBC Sink connector error consuming debezium(oracle) CDC event
I have below images running in docker.
quay.io/debezium/connect:2.0
quay.io/debezium/kafka:2.0
quay.io/debezium/zookeeper:2.0
container-registry.oracle.com/database/enterprise:latest
debezium-source-...
1
vote
1
answer
109
views
Debezium: Produce messages only upon changes to columns in column.include.list
I am running Debezium with column.include.list configured to a subset of columns on each of the observed tables of the source MySQL database. Changes to records on the source tables are being ...
0
votes
0
answers
95
views
Debezium misssing some events
During the process of data discrepancy, I realised some events are missed and not written to kafka during CDC. These are the steps I did:
1- Setup two different debezium connector each fetch data ...
0
votes
0
answers
29
views
Get tenant_id from a delete CDC operation in Debezium
I am trying to develop a SpringBoot application to track CDC changes of a Postgres database.
In DebeziumConfig I have
private io.debezium.config.Configuration config() {
return io.debezium....
0
votes
1
answer
92
views
Debezium is not working with redis streams and postgresql
I'm having an issue with running debezium with redis and postgresql.
My docker compose is:
version: "3.3"
services:
redis-stack:
image: redis/redis-stack:7.0.6-RC4
restart: unless-...
0
votes
0
answers
75
views
Kafka Connect JdbcSinkConnector - error on INSERT (invalid byte sequence)
We have data replication solution using Kafka Connect. Data is being read using Debezium SqlServerConnector into multiple topics, and then written to PostgreSQL using JdbcSinkConnector. Each topic/...
0
votes
1
answer
43
views
Errors with BigQuery Sink Connector Configuration
I am trying to ingest data from MySQL to BigQuery. I am using Debezium components running on Docker for this purpose.
Anytime I try to deploy the BigQuery sink connector to Kafka connect, I am getting ...
0
votes
1
answer
86
views
Confluent connector tasks have failed
Started the docker containers (cp-kafka-connect-base:7.0.1) and installed the self-managed connector (debezium-connector-mysql:latest).
Docker containers works good.
When I am trying to configure the ...
0
votes
0
answers
69
views
Debezium fails parsing a RENAME query
I deployed a Debezium MySQL connector to capture changes for table A (database has many more tables). I see the following exception:
[Worker-0b2620d26c36f7ef3] io.debezium.text.ParsingException: ...
0
votes
0
answers
62
views
Sink Connector auto create tables with proper data type
I have the debezium source connector for Postgresql with the value convertor as Avro and it uses the schema registry.
Source DDL:
Table ...
0
votes
0
answers
44
views
Updating Debezium MySQL connector to resume with database moved to RDS
I have moved my MySQL database to RDS.
I would like to update the connector that publishes messages from an outbox table to a specific Kafka topic to point to the RDS instance.
What would be the steps ...