Why architecture matters here
CDC architecture matters because it's the difference between minutes-fresh analytics and hours-fresh. Real CDC over WAL/binlog is near-zero source impact and sub-second latency; polling is expensive and stale.
Cost is proportional to change volume + downstream count. Kafka + connectors scale cheaply. Managed CDC (Fivetran) charges per source but saves engineering.
Reliability depends on connector state management. If offset lost, you resync or miss changes. Proper checkpointing is essential.
The architecture: every piece explained
Walk the diagram top to bottom.
Source DB. Postgres (WAL), MySQL (binlog), MongoDB (oplog), Oracle (redo log). Any transactional store with a replication log.
CDC Connector. Debezium reads replication log as a replication slave would. Emits structured events.
Kafka Topic. One topic per table (typical). Row events queued for downstream.
Read replication log. The connector reads log positions; commits offsets. Source DB unaffected beyond adding a slot/replica.
Row event schema. {before: {...old row...}, after: {...new row...}, op: "u"/"i"/"d", ts_ms, source metadata}.
Sinks. Data warehouse (Snowflake, BigQuery), data lake (Iceberg/Delta), cache invalidation, search index, other services.
Exactly-once. Kafka Connect exactly-once with proper sink implementation.
Snapshotting. Initial: connector reads full table before switching to log. Incremental snapshotting (Debezium 2.x) refreshes without full re-scan.
Schema Evolution. ALTER TABLE propagates via DDL events; downstream applies.
Backpressure + Lag. Monitor connector lag; alert on backlog growth.
End-to-end CDC flow
Trace a flow. Postgres has a customers table. Enable logical replication; create a Debezium slot.
Debezium connector starts. Snapshot: reads full customers table, emits one event per row to Kafka topic "customers-cdc". Marks snapshot complete.
Switches to WAL streaming. Every INSERT/UPDATE/DELETE emits event within milliseconds.
Downstream: warehouse sink consumes topic, MERGES into destination table using primary key. Data lag: <5 seconds.
Search index sink consumes same topic; upserts into Elasticsearch. Cache invalidation sink watches; on customer update, invalidates that user's cache entries.
Application does ALTER TABLE add column. Debezium detects DDL; emits schema change event. Sinks handle: warehouse alters destination; index adds mapping; app knows about new column.
Connector fails; restart. Reads its saved offset from Kafka; resumes streaming from WAL position. No data loss.