Why architecture matters here
Kafka architecture matters because Kafka's semantic guarantees look simple ("append-only log") but hide considerable engineering to make real. Ordering is per-partition, not per-topic. Exactly-once requires transactions plus idempotent producers plus specific consumer patterns. Replication is quorum-based but tunable via min.insync.replicas. Get these wrong and you get duplicates, gaps, or slow rebalances that read as broken.
Cost matters. Kafka storage grows with retention; tiered storage cuts hot-storage cost by orders of magnitude. Bandwidth between AZs is real money. Batching, compression, and quotas together move the needle by 30-50%.
Reliability is where Kafka is opinionated. Under-provisioned brokers, misconfigured min.insync.replicas, and greedy consumer configurations show up as production incidents. Understanding the failure modes prevents them.
The architecture: every layer explained
Walk the diagram top to bottom.
Producer. Publishes events to topics. Modern producers batch and compress; idempotent producers avoid duplicates on retry; transactional producers give exactly-once when paired with correctly-configured consumers.
Kafka Cluster. A set of brokers managed by KRaft (Kafka's built-in Raft-based metadata quorum replacing ZooKeeper). Broker count typically 5-30; partitions distributed across brokers.
Consumer Group. A set of consumer instances that collectively read all messages from a topic exactly once per group. The group coordinator assigns partitions to consumers; adding or removing consumers triggers a rebalance.
Topic + Partitions. A topic is a log; a partition is an ordered subset of that log. Ordering is guaranteed within a partition; across partitions, only causal ordering per key. Partition count is a critical tuning decision: too few limits parallelism, too many wastes resources.
Replication (ISR). Each partition has a leader and followers. Followers pull from the leader continuously. The In-Sync Replica set is followers within a lag threshold. Producers can require acks from all ISR (acks=all) for durability.
Broker Storage. Each broker stores its assigned partitions as append-only log segments with sparse indexes. Old segments are deleted or shipped to tiered storage based on retention policy.
Schema Registry. Confluent Schema Registry (or alternatives). Stores versioned schemas per topic. Producers embed schema ID; consumers look up the schema to deserialize. Compatibility rules prevent breaking changes.
Kafka Connect. A framework for source and sink connectors — DB CDC in, S3 or warehouse sinks out. Manages worker distribution and offset tracking. Alternative: Debezium for CDC specifically.
Kafka Streams / Flink. Stateful stream processing. Streams is Kafka-native with local RocksDB stores; Flink is more general and horizontally scalable. Both support windowing, joins, and stateful aggregations.
Exactly-Once Semantics. Idempotent producer + transactional writes + read_committed consumer isolation level. When configured together, gives exactly-once processing end to end for Kafka Streams and equivalent Flink checkpointed jobs.
Ops tools. MirrorMaker for cross-cluster replication. Tiered Storage for cheap warm storage. Cruise Control for automated partition balancing.
End-to-end event flow
Trace an event. An order-service producer emits an OrderPlaced event. The producer serializes it as Avro, looks up the schema in the Registry, batches with other events, compresses (LZ4), and sends to the broker.
The broker hosting the leader for the target partition writes the batch to disk, replicates to followers, and acknowledges when all ISR have replicated (with acks=all).
Downstream consumers subscribe. A billing service consumes to charge; an analytics service consumes to update dashboards; a fulfillment service consumes to trigger warehouse tasks. Each consumer group tracks its own offset; each independently reads at its own pace.
The billing consumer processes the event, calls Stripe, records the charge in its DB, and commits its Kafka offset. If a crash happens between charging Stripe and committing offset, on restart it will re-read the event; idempotency via Stripe idempotency-key prevents double-charging.
The analytics service uses Kafka Streams to aggregate hourly revenue by category. It maintains state in local RocksDB; state is durably backed by an internal changelog topic. On rebalance, state migrates via changelog replay.
A month later, retention removes the old segments. Tiered storage kept them in S3; a replay job can reprocess from S3 if needed.