Why architecture matters here
Distributed logs face an unavoidable tension: the more machines that must confirm a write before you acknowledge it, the more durable the write but the higher its latency and the more fragile its availability - one slow replica can stall everything. Kafka's ISR design is an elegant answer to this tension because it makes the durability set adaptive rather than fixed. You do not wait for a fixed quorum of a fixed membership; you wait for whoever is currently keeping up. A healthy follower is in the set and its acknowledgement counts; a sick follower is ejected so it cannot hold up commits. The architecture matters because this one mechanism lets a single system be tuned anywhere along the durability/availability curve.
The consequences are concrete and they are where teams get burned. With acks=all and min.insync.replicas=2 on a three-replica topic, you can lose one broker and keep taking writes with zero data loss - the ISR drops to two, still at the floor, and the leader keeps committing. Lose a second broker and the ISR hits one, below the floor: the leader now rejects producer writes rather than accept data only one machine holds. That rejection looks like an outage, and to an unprepared team it is a surprise - but it is the system correctly refusing to make a durability promise it cannot keep. The alternative configurations fail differently: acks=1 acknowledges as soon as the leader writes, so a leader crash before followers replicate loses acknowledged records silently; and permitting unclean election means a partition can come back online having rewound to an old offset, dropping every write the failed leader had committed.
So the ISR is not an implementation detail - it is the surface where your durability SLA is actually decided. A team that says 'we use Kafka, it's durable' without pinning acks, min.insync.replicas, replication factor, and unclean-election policy has not chosen a durability guarantee; it has inherited whatever the defaults imply, which on some versions has meant silent data loss on failover.
Getting the model right also shapes capacity planning: replication is real network and disk traffic (each write lands on every replica), rack-aware placement decides whether one rack failure can take a partition below its floor, and replication throttling exists precisely because a catching-up replica can otherwise saturate the network and starve live traffic.
The architecture: every piece explained
Top row: the write path. A producer with acks=all sends a batch to the partition's leader replica, which owns the authoritative log and appends the records to its local segment. Followers run a continuous fetch loop against the leader - they are just special consumers pulling the log - and append what they receive to their own copies, then report their fetch offset back. The ISR set is the leader's live accounting of which followers are caught up: a follower stays in as long as it has fetched up to the leader's log end within replica.lag.time.max.ms; fall behind that window and the leader ejects it from the ISR.
Middle row: what 'committed' means. The high watermark is the minimum log-end offset across all ISR members - the highest offset the leader knows every in-sync replica has. The leader only advances the high watermark, and only acknowledges acks=all producers, once the whole ISR has the record; consumers may read only up to the high watermark, so they never see a record that could still be lost. min.insync.replicas is the enforced floor: if the ISR size drops below it, the leader returns NotEnoughReplicas to acks=all producers instead of committing. The leader epoch is a monotonic term number attached to every leadership; followers use it to detect and truncate divergent suffixes, and a returning old leader is fenced because its epoch is stale.
Bottom rows: coordination and the hard choice. The controller - a broker elected via the KRaft metadata quorum (or ZooKeeper in older clusters) - monitors broker liveness and, when a leader dies, elects a new leader from the ISR and propagates the new leader and epoch to all replicas. Unclean leader election is the escape hatch: if no ISR member survives, enabling it lets the controller promote an out-of-sync replica, restoring availability but discarding every record that replica was missing - a deliberate data-loss-for-availability trade. KRaft/ZooKeeper holds the cluster metadata and membership that all of this depends on. The ops strip is the daily signal set: replica lag, under-replicated-partition count, rack-aware placement, and replication throttling.
End-to-end flow
Follow a write through a three-replica partition (brokers B1 leader, B2, B3 followers) with acks=all and min.insync.replicas=2. A producer sends a batch ending at offset 1000. B1 appends it to its log at offsets 995-1000. B2 and B3, in their fetch loops, pull those records and append them, then their next fetch requests report offset 1001 as their position. The leader sees all three ISR members now hold through 1000, advances the high watermark to 1000, and acknowledges the producer. Only now can a consumer read offset 1000.
Now B3 pauses - a long GC. Its fetches stall. After replica.lag.time.max.ms the leader ejects B3 from the ISR, which shrinks to {B1, B2}. That is still at the floor of 2, so the leader keeps committing: new writes are acknowledged once B1 and B2 hold them, and the high watermark advances on the two-member set. When B3's GC ends, it resumes fetching, races to catch up to the leader's log end, and once within the lag window the leader re-admits it; the ISR expands back to three. The under-replicated-partitions metric spiked to 1 during the window and returned to 0 - exactly the signal an operator watches.
Now B1, the leader, crashes. The controller detects the failure and must elect a new leader. Because the ISR was {B1, B2, B3} (or at least contained B2), the controller elects B2 - a fully in-sync replica - as the new leader, assigns it the next leader epoch, and tells B3 and the returning B1 to follow B2 at that epoch. B2 already holds every committed record up to the last high watermark, so no acknowledged data is lost. When B1 restarts, it discovers a higher leader epoch, truncates any records past the high watermark that it had appended but that were never committed cluster-wide (its own uncommitted tail), and re-replicates from B2. The leader epoch is what makes this truncation safe and deterministic rather than a source of divergence.
Finally the catastrophe. B2 and B3 both fail while B1 is still down; the partition's ISR is empty and no in-sync replica exists. With unclean election disabled, the partition goes offline - it will not accept writes or serve reads until an ISR member returns, preserving every committed record at the cost of availability. With unclean election enabled, the controller promotes whichever replica comes back first even if it was behind; the partition is available again immediately, but any records committed after that replica's last fetch are gone, silently. That single toggle is the difference between a durability system and an availability system, and it must be chosen per-topic on purpose.
Two more details make the model complete. First, consumer visibility: throughout all of this, consumers of the partition only ever read up to the high watermark, so during the window when B3 was behind or B2 had just taken over, no consumer could observe a record that was not yet held by the full ISR - readers never see data that a failover could later erase. This is why the high watermark, not the leader's log end, is the boundary of consumable data. Second, catch-up cost: when B1 rejoins after its crash it must re-replicate everything it missed, and that catch-up fetch competes with live produce/consume traffic for the leader's network and disk. On a busy partition an unthrottled catch-up can itself push other followers out of the ISR, so the replication throttle exists to cap recovery bandwidth - recovering a broker is a scheduled resource cost, not a free background task.
Finally, note how min.insync.replicas and replication factor compose into a fault-tolerance budget you can state precisely. RF=3, min.insync=2 tolerates one broker failure with continued writes and zero acknowledged-data loss, and a second failure with read-only availability (writes rejected) but still no loss. RF=3, min.insync=3 tolerates zero failures for writes (any one down stops acks=all) but maximizes durability. RF=5, min.insync=3 tolerates two failures with continued writes. The right point on this curve is a per-topic decision driven by how much the data is worth and how much write-availability you need during failures - and it is a decision, not a default to inherit.