Apache Kafka is often summarised as an append-only log, which is accurate and not much help when something goes wrong. A production Kafka deployment consists of brokers that store partitioned logs, a controller quorum that decides which broker leads each partition, a replication protocol that decides when a write is durable, clients that batch and track offsets, and an ecosystem around them: schema registry, Connect, stream processors, mirroring and tiered storage. The guarantees users rely on, such as ordering, durability and exactly-once, each come from a specific mechanism in that stack, and each fails in a specific way when misconfigured.
This article follows one record through the system: the producer, the leader's log, the followers, the high watermark, the consumer group and the control plane that coordinates them. It then sizes a cluster and walks through a broker failure at peak. For a gentler introduction, read the Kafka overview first; this page is the architecture view.
The moving parts and their jobs
| Component | Job | What breaks if it is wrong |
|---|---|---|
| Broker | Stores partition replicas, serves produce and fetch requests | Disk or network saturation shows up as latency and ISR shrink |
| KRaft controllers | Raft quorum holding cluster metadata; elect partition leaders | Losing quorum freezes metadata changes such as leader election |
| Topic and partition | A topic is a set of partitions; each partition is one ordered log | Too few limits parallelism; too many costs memory, files and failover time |
| Replica set and ISR | Leader plus followers; the ISR is the replicas currently caught up | Weak settings acknowledge writes that a failure can lose |
| Producer | Partitions, batches, compresses, retries | Bad keys create hot partitions; unsafe retries reorder or duplicate |
| Consumer group | Members split the partitions and commit offsets | Slow processing triggers rebalances; early commits lose records |
| Ecosystem | Schema Registry, Connect, Streams or Flink, MirrorMaker 2 | Incompatible schemas or unscoped exactly-once claims |
Since Kafka 4.0, released in March 2025, KRaft is the only metadata mode; ZooKeeper support has been removed. Clusters still on ZooKeeper have to migrate on a 3.x release before upgrading. See KRaft versus ZooKeeper for the migration.
The moving parts and their jobs
| Component | Job | What breaks if it is wrong |
|---|---|---|
| Broker | Stores partition replicas, serves produce and fetch requests | Disk or network saturation shows up as latency and ISR shrink |
| KRaft controllers | Raft quorum holding cluster metadata; elect partition leaders | Losing quorum freezes metadata changes such as leader election |
| Topic and partition | A topic is a set of partitions; each partition is one ordered log | Too few limits parallelism; too many costs memory, files and failover time |
| Replica set and ISR | Leader plus followers; the ISR is the replicas currently caught up | Weak settings acknowledge writes that a failure can lose |
| Producer | Partitions, batches, compresses, retries | Bad keys create hot partitions; unsafe retries reorder or duplicate |
| Consumer group | Members split the partitions and commit offsets | Slow processing triggers rebalances; early commits lose records |
| Ecosystem | Schema Registry, Connect, Streams or Flink, MirrorMaker 2 | Incompatible schemas or unscoped exactly-once claims |
Since Kafka 4.0, released in March 2025, KRaft is the only metadata mode; ZooKeeper support has been removed. Clusters still on ZooKeeper have to migrate on a 3.x release before upgrading. See KRaft versus ZooKeeper for the migration.
The write path: from send() to acknowledged
When an application calls send(), the producer serialises the key and value. If you use a schema registry, it writes the schema id into the payload. Then the partitioner picks a partition. A keyed record goes to the partition chosen by hashing the key, so every record for one key lands in one partition and stays in order. Records without a key are spread across partitions in sticky batches. The record joins an in-memory batch for that partition. A batch is sent when it reaches batch.size or when linger.ms expires, and it is compressed as a unit, which is why batching and compression together drive throughput.
The batch goes to the partition's leader broker. The leader appends it to the active log segment and assigns offsets. What happens next depends on acks. With acks=0 the producer does not wait. With acks=1 the leader replies once the batch is in its own log, so a leader crash before replication can lose it. With acks=all, the default since Kafka 3.0, the leader replies only once every replica in the current ISR has the batch.
Idempotence, also on by default since 3.0, gives each producer an id and each batch a sequence number per partition. A broker that sees a duplicate sequence from a retry discards it, so retries do not create duplicates or reorder records within a partition.
# producer.properties: durable, ordered, reasonably efficient
acks=all
enable.idempotence=true
compression.type=zstd
linger.ms=10
batch.size=131072
delivery.timeout.ms=120000 # total time to keep retrying a record
# topic: three replicas, and a write needs at least two in sync to succeed
kafka-topics.sh --bootstrap-server b1:9092 --create --topic orders \
--partitions 48 --replication-factor 3 --config min.insync.replicas=2Replication, the ISR and the high watermark
Followers replicate by fetching from the leader, exactly as consumers do. Each replica tracks its log end offset, the next offset it will write. A follower stays in the in-sync replica set while it keeps up. If it has not caught up to the leader's end within replica.lag.time.max.ms (30 seconds by default), the leader removes it from the ISR, and the controller records the change.
The high watermark is the highest offset that every ISR member has. Consumers using the default read_uncommitted isolation level can read only up to the high watermark, so they never see a record that a leader failover could erase. Transactional consumers using read_committed stop earlier, at the last stable offset, which excludes transactions that are still open.
The ISR shrinks under load, so acks=all by itself is not enough. If the ISR has shrunk to just the leader, acks=all is satisfied by the leader alone. The setting min.insync.replicas closes that gap. With a replication factor of 3 and min.insync.replicas=2, a write succeeds only if at least two replicas have it, and otherwise the producer gets NotEnoughReplicas and retries. The broker default is 1, so set it explicitly. Keep unclean.leader.election.enable at its default of false, so an out-of-sync replica can never become leader and silently drop acknowledged writes. Leader epochs let a returning replica truncate any divergent suffix before it rejoins. ISR and replication goes deeper.
Storage: segments, retention and tiers
On disk, each partition replica is a directory of segments. Each segment is a log file with an offset index and a time index. Only the newest segment receives appends. Reads use the sparse offset index to find a position, then stream bytes, often straight from the page cache. Segments roll at segment.bytes (1 GiB by default) or after segment.ms, and retention deletes whole closed segments once they exceed retention.ms or retention.bytes. Compacted topics instead keep the latest record per key, which suits changelog and table-like data.
Tiered storage, production-ready since Kafka 3.9, ships closed segments to an object store through a pluggable remote storage manager and keeps only a recent local window on broker disks, set with local.retention.ms or local.retention.bytes. Consumers reading old offsets fetch from the remote tier transparently. The architectural gain is that broker disk is sized for the hot window rather than full retention, and moving or replacing a broker copies far less data.
The read path: consumer groups and offsets
Consumers in a group divide a topic's partitions among themselves. Each partition is read by at most one member at a time, so parallelism per group is capped at the partition count. A consumer fetches from the leader by default, or from a nearby follower if rack-aware fetching is configured, processes records and commits offsets to the internal __consumer_offsets topic. Delivery semantics come from the order of processing and commit. Commit after processing and you get at-least-once, which means duplicates after a crash, so make processing idempotent. Commit before processing and you get at-most-once. Exactly-once needs transactions, covered below.
Membership changes trigger a rebalance. The group protocol from KIP-848, generally available in Kafka 4.0, moves assignment to the broker-side group coordinator and reconciles incrementally, so members that keep their partitions are not paused while others move. Consumers opt in with group.protocol=consumer. Under the older protocol, cooperative-sticky assignment gives a similar effect. Either way, keep each poll loop's processing time well inside max.poll.interval.ms, or the member is evicted and its partitions are reassigned while it is still working. See Kafka consumers.
Exactly-once, and where it stops
Kafka's exactly-once is precise in scope: a consume-transform-produce loop whose input and output are both Kafka topics. A transactional producer writes output records and the consumer's offsets in one transaction. Downstream read_committed consumers see either all of it or none. Kafka Streams turns this on with processing.guarantee=exactly_once_v2, and Flink achieves the equivalent with checkpoints and a two-phase-commit Kafka sink.
Once a record leaves Kafka, for example to a database, an email service or a payment API, the guarantee stops at the boundary. The sink then needs idempotent writes keyed on a stable id, or its own transaction that stores the consumed offset alongside the data. Treat any end-to-end exactly-once claim that does not explain the sink with suspicion.
Worked example: sizing a cluster and losing a broker at peak
An order platform ingests 150 MB/s of compressed records at peak, 60 MB/s on average. It keeps seven days of data, has two consumer groups reading everything, uses a replication factor of 3 across three availability zones, and runs min.insync.replicas=2.
- Disk writes at peak: 150 MB/s times 3 replicas gives 450 MB/s across the cluster.
- Network: 150 MB/s in from producers, 300 MB/s of follower fetches, and 300 MB/s out to two consumer groups. Most replication traffic crosses zones, which matters for cloud bills.
- Storage: 60 MB/s times 86,400 s times 7 days is about 36 TB per replica copy, or about 109 TB with three replicas. On nine brokers that is about 12 TB each before headroom.
- With tiered storage: keeping 24 hours locally needs about 15.6 TB of local disk across the cluster, under 2 TB per broker. The object store holds one copy of the rest.
Nine brokers keep peak write load near 50 MB/s each, which leaves room for a failure. Now broker 4 loses its disk at peak. It leads about a ninth of the partitions. Within seconds the controllers notice the missed heartbeats, fence the broker, and elect new leaders from each affected partition's ISR. Producers get NOT_LEADER_OR_FOLLOWER errors, refresh metadata and retry, and idempotence keeps those retries free of duplicates. Affected partitions now have two in-sync replicas, which still satisfies min.insync.replicas=2, so writes continue. The eight survivors take about 12.5 percent more traffic each. Consumers of moved partitions see a brief pause and resume from their committed offsets.
The risk window is what comes next. A second failure in the same partitions would leave one in-sync replica and block writes to them, which is correct behaviour since the alternative is silent loss. The team's runbook therefore replaces the broker immediately. With tiered storage, it re-replicates only the local window rather than 12 TB, and it throttles reassignment so recovery traffic does not starve producers.
Failure modes and trade-offs
| Failure mode | Symptom | Prevention |
|---|---|---|
| Acknowledged write lost | Gap after leader failover | acks=all, min.insync.replicas=2 with RF 3, no unclean election |
| Hot partition | One broker saturated, lag on one partition | Better key choice; see partitioning strategies |
| Rebalance storms | Repeated pauses, lag sawtooth | KIP-848 protocol or cooperative-sticky, shorter processing per poll |
| ISR flapping | UnderReplicatedPartitions alerts under load | Fix disk or network headroom before tuning lag thresholds |
| Controller quorum loss | No leader elections or topic changes | Three or five controllers in separate failure domains |
| Breaking schema change | Consumers fail to deserialize | Registry compatibility mode enforced in CI |
| Duplicates in a sink | Double-applied side effects | Idempotent sink keyed on a record id, or transactional sink |
The core trade-offs are durability against latency (acks=all adds a replication round trip), partition count against overhead and recovery time, and retention against cost, which tiered storage largely decouples. Cross-zone replication buys availability at a real network cost. Operate Kafka as a managed service unless you have the people to own these decisions.
What to do next
- For every topic, verify replication factor 3,
min.insync.replicas=2and unclean leader election disabled. - Confirm producers use
acks=allwith idempotence on, and tunelinger.ms,batch.sizeand compression against your throughput. - Alert on under-replicated partitions, offline partitions, ISR shrink rate, consumer lag per partition and request latency percentiles.
- Size disk, network and broker count with the arithmetic above, and leave room to lose one broker at peak.
- Plan the move to KRaft-only 4.x and the KIP-848 consumer protocol if you are on older releases.
- Evaluate tiered storage if retention drives broker disk size.
- Write down where exactly-once ends for each pipeline, and make those sinks idempotent; Kafka exactly-once has the patterns.