5

I wonder if it is possible, or if someone has tried to setup Apache Kafka as consumer of PostgreSQL logigal log stream? Does that even makes sense?

https://wiki.postgresql.org/wiki/Logical_Log_Streaming_Replication

I have a legacy source system that I need to make realtime dashboard from. For some reasons I can't hook the application events (btw, it's java app). Instead, I'm thinking of some kind of a lambda architecture: when dashboard initializes, it reads from persisted data "data warehouse" which gets there after some ETL. And then changing events are streamed via Kafka to the the dashboard.

Another use of the events stored in Kafka would be a kind of change data capture approach for data warehouse population. This is necessary because there is no commercial CDC tool that supports postgesql. And the source application is updating tables without keeping history.

4
  • I might be easier to create triggers on postgres tables and use this triggers to push changes to other systems. BTW Old postgres replication systems used similar setup with triggers to do replication. Oct 7, 2014 at 8:59
  • Igor, thanks! We fear that triggers would affect performance of the system (we'd forward all UPDATE, DELETE statements to shadow tables).
    – Andrew
    Oct 7, 2014 at 10:37
  • Another option is to builds dashboards on postgres streaming replication slave. The limitation - this kind or replication permits only read only requests on slave. Oct 7, 2014 at 12:44
  • Igor, we do that already, however this isn't event driven and not "realtime" - because we have to select data periodically from replica and we have 2 problems: sometimes it lags badly, sometimes query execution time is not fast enough.
    – Andrew
    Oct 7, 2014 at 13:45

3 Answers 3

2

A combination of xsteven's PostgreSQL WAL to protobuf project - decoderbufs (https://github.com/xstevens/decoderbufs) - and his pg_kafka producer (https://github.com/xstevens/pg_kafka) might be a start.

0
1

Take a look at Bottled Water which:

uses the logical decoding feature (introduced in PostgreSQL 9.4) to extract a consistent snapshot and a continuous stream of change events from a database. The data is extracted at a row level, and encoded using Avro. A client program connects to your database, extracts this data, and relays it to Kafka

They also have Docker images so looks like it'd be easy to try it out.

1

The Debezium project provides a CDC connector for streaming data changes from Postgres into Apache Kafka. Currently it supports Decoderbufs and wal2json as logical decoding plug-ins. Bottled Water referenced in Steve's answer is comparable, but it is not actively maintained any longer.

Disclaimer: I'm the project lead of Debezium

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.