All 17 articles, sorted alphabetically
Kafka Partitioning Strategies
Pick the key wrong, regret it later.
Read article →Backpressure architecture
Deep-dive on backpressure in stream processing: why unbounded buffering causes stateful crashes, credit-based flow control between operator subtasks, …
Read article →Kafka consumer group rebalancing architecture
Deep-dive on Kafka consumer group rebalancing: the group coordinator and heartbeat liveness, revoke-and-assign phases, offset commits at the rebalance…
Read article →Exactly-once stream processing architecture
Deep-dive on end-to-end exactly-once semantics: replayable Kafka sources, Flink barrier-aligned snapshots, two-phase-commit sinks bound to checkpoint …
Read article →Apache Flink Architecture in Depth
A 2500-word walkthrough of Apache Flink architecture: JobManager, TaskManagers, dataflow graph, state backends, sources/sinks, windows, watermarks, ch…
Read article →Flink Async I/O architecture
Deep-dive on Flink Async I/O: overlapping many in-flight external lookups through a bounded-capacity buffer, the AsyncFunction and ResultFuture contra…
Read article →Flink savepoints architecture
Deep-dive on Apache Flink savepoints: user-triggered canonical-format state snapshots keyed by stable operator UIDs, exactly-once continuity across up…
Read article →Apache Kafka Streaming Architecture in Depth
A 2500-word walkthrough of Apache Kafka: producers, brokers with KRaft, topics + partitions, replication, schema registry, connect, stream processing,…
Read article →Kafka ISR replication architecture - leaders, followers, and the in-sync replica set
Deep-dive on Kafka's replication protocol: leader and follower fetch, the in-sync replica set, high watermark and commit sema…
Read article →Kafka client quotas architecture
Deep-dive on Kafka quotas: user/client-id entity precedence, produce/fetch/request/mutation quota types, the sliding-window rate sampler, the throttle…
Read article →Kafka Streams
Deep-dive on Kafka Streams: processor topologies over KStream/KTable, local RocksDB state stores backed by changelog topics, partitioning and stream t…
Read article →Kafka log compaction architecture
Deep-dive on Kafka log compaction: clean vs dirty segments, the cleaner thread and its offset map, tombstones and delete.retention.ms, dirty-ratio and…
Read article →Kafka MirrorMaker 2 architecture
Deep-dive on Kafka MirrorMaker 2 (MM2): cross-cluster replication built on Kafka Connect that copies records, topic configs, ACLs, and consumer offset…
Read article →Schema registry architecture
Deep-dive on the streaming schema registry: subjects and versions, the schema-id wire format, backward/forward/full compatibility modes, reader vs wri…
Read article →Stateful stream joins -- joining unbounded streams
Deep-dive on stateful stream joins: the unbounded-join problem, keeping both sides as state (buffered by key), window/interval joins bounding the stat…
Read article →Kafka tiered storage architecture
Deep-dive on Kafka tiered storage: the RemoteLogManager and remote index, segment rolling and upload, local vs remote retention, tail vs backfill read…
Read article →Stream windowing -- bounding the unbounded
Deep-dive on stream windowing: the unbounded-stream aggregation problem, tumbling/sliding/session windows, event vs processing time, watermarks for ev…
Read article →