All 46 articles, sorted alphabetically
Change Data Capture in 2026, in depth: choosing a topology, surviving primary failover with Postgres 17 failover slots, schema evolution and landing changes in the lakehouse
The decisions behind a CDC pipeline in 2026: Debezium on Kafka Connect, Debezium Server, Flink CDC pipelines or an embedded engine; Postgres 17 failov…
Read article →Debezium CDC to Kafka, in depth: connectors, the change-event envelope, snapshots, signals, SMTs and exactly-once
A practical guide to running Debezium on Kafka Connect: deploying PostgreSQL and MySQL connectors, reading the change-event envelope, topic naming, sc…
Read article →Event Sourcing vs Event Streaming, in depth: system of record versus integration log, concurrency, and how to bridge them
Event sourcing and event streaming both store events but solve different problems. How an event store enforces per-stream optimistic concurrency, why …
Read article →Exactly-Once Semantics in Streaming, in depth: idempotent producers, Kafka transactions, fencing and idempotent sinks
What exactly-once really guarantees in a streaming pipeline and how it is built: replayable input, the Kafka idempotent producer, transactions, the la…
Read article →Flink Checkpoint Design, in depth: sizing interval, timeout and min-pause from a recovery budget, aligned versus unaligned checkpoints, sink commit latency and reading checkpoint metrics
How to design Flink checkpointing for a real job rather than accept defaults: what a checkpoint costs and where time goes, deriving the interval from …
Read article →Flink State Backends Compared, in depth: hashmap vs RocksDB vs ForSt for latency, checkpoints, recovery, memory and migration
An operational comparison of Flink's state backends in Flink 2.x: what hashmap, RocksDB and the experimental ForSt backend cost per state access,…
Read article →Kafka KRaft vs ZooKeeper, in depth: the metadata plane, the controller quorum, fencing and a migration you can only finish once
How Kafka's metadata plane changed from ZooKeeper to KRaft: what the old controller did and why it hit limits, the Raft-replicated metadata log, …
Read article →Kafka Partitioning Strategies
Kafka partitioning strategies in depth: choosing keys, partition counts, co-partitioning for joins, consumer assignment, a 12-to-48 partition migratio…
Read article →Kafka vs Pulsar vs Redpanda, in depth: storage models, what an acknowledged write means, scaling cost, consumption models and how to choose
A practical comparison of Apache Kafka, Apache Pulsar and Redpanda: where data lives, replication and fsync defaults, rebalancing cost, consumer group…
Read article →Materialize for Streaming SQL, in depth: differential dataflow, sources, indexes vs materialized views, SUBSCRIBE and operating clusters
How Materialize keeps SQL views up to date as data changes: the storage, compute and adapter layers, the (row, time, diff) update model behind differe…
Read article →Redpanda Tiered Storage, in depth: what gets uploaded and when, two retention layers, chunked remote reads, sizing, recovery and read replicas
How Redpanda Tiered Storage works and how to run it: the upload path from Raft segments to object storage, local versus total retention, chunked reads…
Read article →Schema Registry Patterns, in depth: subject naming, CI-owned registration, compatibility per subject, safe evolution and breaking-change migrations
Operating patterns for a Kafka schema registry: choosing a subject naming strategy and handling multi-event topics, registering schemas in CI instead …
Read article →Apache Flink Overview, in depth: the model in one picture, choosing SQL or DataStream, one job written both ways, deployment, upgrades and the 2.x migration
A practical on-ramp to Apache Flink: streams, keyed state, event time and snapshots explained from first principles, how to choose between Flink SQL, …
Read article →Flink State, in depth: writing stateful functions with timers, what each access costs, testing with the harness and reading state from savepoints
A practical guide to programming Flink state: the current-key contract, a worked KeyedProcessFunction with MapState and deduplicated timers, the per-a…
Read article →Flink Windowing, in depth: assigners, triggers, lateness, incremental aggregation, session merging and window TVFs, with a worked event-time timeline
How Apache Flink windows really work: the keyed window operator, tumbling, sliding, session and global assigners, how watermarks fire windows, increme…
Read article →Kafka Consumers and Consumer Groups, in depth: the poll loop, fetching, offset commits, classic versus KIP-848 group protocols, threading models and lag
How Kafka consumers really work: what poll() does, the fetch settings that matter, auto and manual offset commits and their delivery guarantees, the c…
Read article →Kafka Exactly-Once Semantics, in depth: the broker side, from producer state and the transaction coordinator to the last stable offset, hanging transactions and KIP-890
How Kafka brokers implement exactly-once: per-partition producer state and sequence checks, the transaction coordinator and __transaction_state, contr…
Read article →Apache Kafka Overview, in depth: the log model, partitions and ordering, producer and consumer settings that matter, sizing a first topic and the failure modes to plan for
A first-principles overview of Apache Kafka for engineers about to run it: the partitioned append-only log, offsets and consumer groups, keys and per-…
Read article →Kafka Partitions, in depth: how the producer chooses a partition, per-partition batching, hot keys and skew, and cross-client hashing
A producer-side guide to Kafka partitions: the Java client's partition choice rules and their history, the sticky and adaptive behaviour for unke…
Read article →Apache Pulsar, in depth: the client side, from producer batching and deduplication to flow control, retries, delayed delivery, schemas and transactions
How Pulsar behaves from the application's side: producer batching defaults and send timeouts, partition routing, idempotent producers and dedupli…
Read article →Spark Streaming, in depth: the micro-batch lifecycle, where a batch spends its time, trigger modes including real-time mode, state stores, transformWithState and choosing Spark or Flink
Spark as a streaming engine: why DStreams gave way to Structured Streaming, the micro-batch lifecycle and its offset and commit logs, a latency budget…
Read article →Stream-table duality, in depth: folds, changelogs, retractions, tombstones and temporal joins
Stream-table duality from first principles: a table as a fold over a stream and a changelog as a table's history, event versus changelog versus r…
Read article →Backpressure architecture
Deep-dive on backpressure in stream processing: why unbounded buffering causes stateful crashes, credit-based flow control between operator subtasks, …
Read article →Apache Beam Architecture, in depth: the unified model, the portability layer, runners, bundles, state and timers
How Apache Beam is built and how a pipeline actually runs: PCollections and PTransforms, the Runner API and Fn API, SDK harnesses, fusion and bundles,…
Read article →CDC Architecture, in depth: running change data capture as a streaming system, from replication slot to upsert sink
Change data capture as a running streaming system: Debezium on Kafka Connect, source offsets and restart semantics, the change envelope as a changelog…
Read article →Kafka consumer group rebalancing architecture
Deep-dive on Kafka consumer group rebalancing: the group coordinator and heartbeat liveness, revoke-and-assign phases, offset commits at the rebalance…
Read article →Exactly-Once Stream Processing Architecture, in depth: the source, snapshot and sink chain
Exactly-once stream processing end to end: replayable sources, barrier snapshots, four sink protocols, Flink, Kafka Streams and Spark, a crash-injecti…
Read article →Apache Flink Architecture in Depth
A 2500-word walkthrough of Apache Flink architecture: JobManager, TaskManagers, dataflow graph, state backends, sources/sinks, windows, watermarks, ch…
Read article →Flink Async I/O Explained: AsyncFunction, Capacity, Ordering
Apache Flink Async I/O for stream enrichment: AsyncFunction and ResultFuture, capacity in flight, ordered vs unordered output, timeouts, retries, exac…
Read article →Flink Savepoints Explained: Operator UIDs, Upgrades, Rescaling
How Apache Flink savepoints work: user-triggered portable state snapshots keyed by stable operator UIDs, restoring after job upgrades, rescaling by ke…
Read article →Flink State Architecture, in depth: keyed and operator state, key groups, heap vs RocksDB, TTL, schema evolution and sizing
How Apache Flink stores and scales state: keyed state primitives and when to use each, operator and broadcast state, key groups and max parallelism, h…
Read article →Apache Hudi architecture, in depth: streaming upserts with Flink and HoodieStreamer, the timeline, MOR compaction and incremental reads
How Apache Hudi works as a streaming sink and source: file groups and slices, the timeline, Flink writes committed on checkpoints, index choice for st…
Read article →Apache Kafka Streaming Architecture in Depth
Kafka architecture in depth: brokers, KRaft, the write path, ISR and high watermark, storage and tiering, consumer groups, scoped exactly-once, cluste…
Read article →Kafka ISR Explained: In-Sync Replicas, High Watermark, acks=all
How Kafka ISR replication works: the in-sync replica set, follower fetch, high watermark, min.insync.replicas with acks=all, leader epochs, unclean el…
Read article →Kafka ordering architecture, in depth: what the log guarantees, the six places order breaks, and how to keep per-key order end to end
A practical guide to message ordering in Apache Kafka: the per-partition guarantee, the idempotent producer's sequence numbers and the settings t…
Read article →Kafka client quotas architecture
Deep-dive on Kafka quotas: user/client-id entity precedence, produce/fetch/request/mutation quota types, the sliding-window rate sampler, the throttle…
Read article →Kafka Streams: State Stores, RocksDB Changelogs, Exactly-Once
How Kafka Streams works: KStream and KTable topologies, RocksDB state stores backed by changelog topics, exactly-once via transactions, standby replic…
Read article →Kafka Log Compaction Explained: Cleaner, Tombstones, Dirty Ratio
How Kafka log compaction keeps the latest value per key: clean vs dirty segments, the cleaner and offset map, tombstones, delete.retention.ms, dirty r…
Read article →Kafka MirrorMaker 2 architecture
Deep-dive on Kafka MirrorMaker 2 (MM2): cross-cluster replication built on Kafka Connect that copies records, topic configs, ACLs, and consumer offset…
Read article →Apache Paimon architecture, in depth: snapshots, LSM buckets, merge engines, changelog producers, compaction and streaming reads
How Apache Paimon stores a streaming-updatable lake table: snapshots, manifests and LSM sorted runs per bucket; fixed, dynamic and postpone bucket mod…
Read article →Apache Pulsar Architecture in Depth: stateless brokers, BookKeeper ledgers, subscriptions and operations
How Apache Pulsar is built and run: stateless brokers that own topic bundles, BookKeeper ledgers with ensemble, write and ack quorums, fencing on fail…
Read article →Redis Streams Architecture, in depth: entry IDs, the listpack radix tree, consumer groups and the PEL, trimming, replication limits and sharding
How Redis Streams work inside and how to run them safely: the millisecond-sequence entry ID, radix tree of listpack nodes, consumer-group state and th…
Read article →Schema Registry Explained: Compatibility Modes, Schema Evolution
How a schema registry keeps Kafka producers and consumers compatible: subjects and versions, the schema ID wire format, backward, forward, full compat…
Read article →Stateful stream joins -- joining unbounded streams
Deep-dive on stateful stream joins: the unbounded-join problem, keeping both sides as state (buffered by key), window/interval joins bounding the stat…
Read article →Kafka tiered storage architecture
Deep-dive on Kafka tiered storage: the RemoteLogManager and remote index, segment rolling and upload, local vs remote retention, tail vs backfill read…
Read article →Stream windowing -- bounding the unbounded
Deep-dive on stream windowing: the unbounded-stream aggregation problem, tumbling/sliding/session windows, event vs processing time, watermarks for ev…
Read article →