Imagine your database as a shared notebook that hundreds of services need to read simultaneously. The naive solution is to ask every service to poll the notebook every few seconds - but with 50 million writes per day, that polling tax compounds into billions of redundant queries, and you still miss events that happened between polls. The better solution is to stop reading the notebook and start reading the author’s hand as they write: tap into the transaction log directly.

Change Data Capture (CDC) is exactly that. Rather than having application code explicitly publish events alongside every database write - the fragile dual-write pattern that creates consistency windows - CDC reads the database’s own internal record of every committed change. Every row insertion, update, and deletion is already logged atomically with the transaction. CDC simply subscribes to that log and converts it into a stream of structured events.

The challenge is not conceptually hard. The engineering is. Transaction logs were designed for crash recovery, not for external consumption. They use database-internal formats, rotate on a schedule, and contain both committed and rolled-back transactions interleaved. A production-grade CDC pipeline must parse these logs reliably, filter only committed data, preserve transactional boundaries across multi-row changes, handle schema changes mid-stream, survive connector crashes without losing or duplicating events, and deliver to downstream consumers that may be temporarily offline. At the scale of a major e-commerce platform - say 200k transactions per minute across 40 tables - each of these problems becomes a hard distributed systems challenge.

This post walks through designing exactly such a pipeline. We will cover the binlog and WAL internals that make it possible, how Debezium sits on top of those logs as a managed connector, the ordering and exactly-once semantics that determine correctness, and how the whole system scales to multiple source databases and thousands of downstream consumers.

Requirements and Constraints

Functional requirements:

• Capture every committed INSERT, UPDATE, and DELETE from MySQL, Postgres, and MongoDB source databases
• Deliver events to downstream consumers in the order they occurred within each table (per-entity ordering)
• Preserve transactional boundaries: all rows changed in one transaction appear as a coherent group
• Support at least 15 downstream consumer types (search indexes, data warehouses, cache invalidators, audit logs)
• Handle schema changes (column additions, renames) without losing events or requiring consumer downtime
• Provide at-least-once delivery with idempotency support so consumers can safely deduplicate

Non-functional requirements:

• End-to-end latency under 500ms from database commit to consumer delivery (p99)
• Zero event loss even if any single component (connector, Kafka broker, consumer) crashes and restarts
• Support backpressure: slow consumers must not cause the pipeline to drop events or block the source database
• Connector throughput sufficient for 50k events/second per source database
• Observable: lag, throughput, and schema drift are all monitorable via standard metrics

Scale assumptions:

• 5 source databases (3 MySQL shards, 1 Postgres cluster, 1 MongoDB replica set)
• Peak write rate: 80k transactions/minute across all sources
• Average event size: 2KB (before + after row image)
• 15 downstream consumers across 4 consumer groups
• Data retention in Kafka: 7 days (enables consumer replay)
• Schema changes: approximately 2 per week per database

Out of scope:

• Global ordering across different source databases (we guarantee per-partition ordering only)
• Exactly-once delivery at the consumer side (we provide at-least-once; consumers handle idempotency)
• Real-time DDL execution on downstream systems

High-Level Architecture

The pipeline splits into three layers. The Source Layer contains the origin databases whose transaction logs we tap. The CDC Layer contains Debezium connectors running as Kafka Connect source connectors, which read those logs and publish structured events to Kafka. The Sink Layer contains downstream consumers - Elasticsearch for search, a data warehouse for analytics, cache invalidators for read performance - each reading from Kafka at their own pace.

Kafka sits at the center as the durable, ordered buffer between producers (Debezium connectors) and consumers. This decoupling is what solves the offline consumer problem: if the Elasticsearch indexer is redeploying for 10 minutes, events accumulate in Kafka and are replayed in order when it comes back. The source database is never asked to hold events on behalf of slow consumers.

The topic naming convention mirrors the source: {server-name}.{database}.{table}. A change to customers.accounts on the mydb MySQL instance lands in the mydb.customers.accounts topic. Consumers subscribe to one or more topics. Each topic has multiple partitions, and events are routed to partitions by primary key hash, guaranteeing that all events for a given row go to the same partition and thus arrive at consumers in order.

The Binlog Reader

The foundation of the entire pipeline is reading the database’s transaction log. Each database engine exposes this differently.

MySQL binary log (binlog) is a sequence of binary event files (e.g., mysql-bin.000042) containing a record of every change that modifies data or schema. There are three binlog formats:

• STATEMENT: logs the SQL statement itself. Cheap to store but non-deterministic - NOW() or UUID() calls produce different results on replay.
• ROW: logs the before and after image of every affected row. Larger but completely deterministic and required for CDC.
• MIXED: uses STATEMENT for safe queries, ROW for non-deterministic ones. CDC tools treat this as unreliable.

We require binlog_format = ROW and binlog_row_image = FULL (captures both before and after images even for partial updates). Debezium registers itself as a MySQL replica, receiving the same binary log stream that a read replica would receive, using the replication protocol.

Postgres Write-Ahead Log (WAL) is fundamentally different. Postgres writes every change to the WAL before applying it to the heap files - this is how it achieves durability. For external CDC, Postgres exposes logical replication slots, which decode the physical WAL entries into logical row-level changes using a decoder plugin. The pgoutput plugin (built into Postgres 10+) and wal2json are the most common choices.

-- Enable logical replication in postgresql.conf
ALTER SYSTEM SET wal_level = 'logical';
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

-- Grant replication privilege to the CDC user
ALTER ROLE debezium_user REPLICATION LOGIN;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium_user;

A replication slot is a cursor into the WAL that Postgres holds open on behalf of a consumer. Crucially, Postgres will not discard WAL segments until all slots have consumed them. This is both a safety feature (events cannot be lost) and a danger: an inactive slot causes WAL to accumulate indefinitely, eventually filling disk.