Why architecture matters here
Replication is asynchronous in practice even when it is 'synchronous' in name: at any instant the leader's log is longer than most followers', because copying takes time. That means the leader holds entries that are not yet on enough replicas to survive its own failure. If a consumer were allowed to read those entries and the leader then crashed, a new leader elected from the followers would not have them — the consumer would have observed data that no longer exists in the system's authoritative history. This is the core hazard the high-water mark exists to prevent: it defines the prefix of the log that is safe to expose because it is present on enough replicas that any future leader is guaranteed to have it too.
The mirror-image problem is storage. An append-only log grows without bound. You cannot keep every entry forever, and you should not force a new or recovering replica to re-download years of history to catch up. So the system periodically snapshots state and reclaims old log segments. But reclaiming too aggressively is dangerous: if you delete an entry that a lagging follower or a slow consumer still needs, that follower can never reconstruct the missing range and is stuck. The low-water mark is the floor that makes reclamation safe — it is the oldest offset the system still promises to keep, and nothing below it may be read or relied upon by anyone.
Three guarantees rest on getting these boundaries right. Read safety: consumers see only committed data, so a failover never retracts a value they already observed. Monotonic commitment: the high-water mark only advances (within a leader term), so once an entry is committed it stays committed — the basis for exactly-once and idempotent processing downstream. Bounded, recoverable storage: the low-water mark lets the system cap disk usage while still guaranteeing that any replica between the low and high marks can be brought current from retained data plus a snapshot.
The cost of watermarks is coordination and honesty. The high-water mark can only advance as fast as the slowest counted replica, so a sick follower drags down the commit boundary and stalls consumers — the mark is a truth-teller about cluster health, which is exactly why you must watch it. And truncation at the low-water mark is irreversible, so the policy that sets it has to be conservative enough to never orphan a replica that could still legitimately catch up.
The architecture: every piece explained
Top row: the logs and their end offsets. The leader log is the append-only source of truth; each entry has a monotonically increasing offset. Every follower maintains a log-end offset (LEO) — the offset just past its last replicated entry. Followers pull (or receive) entries from the leader and advance their LEO as they persist them. Because followers run at different speeds, their LEOs form a spread below the leader's own LEO; the shape of that spread is what the watermarks summarize.
Middle row: the two marks. The high-water mark is computed by the leader as the highest offset that has been replicated to enough followers to count as committed — in a quorum system, the median/minimum of the in-sync replicas' LEOs; in Kafka's ISR model, the minimum LEO across the in-sync replica set. As followers catch up, the leader recomputes and advances the HW; as a follower falls behind (or drops out of the in-sync set), the HW holds or the set shrinks. The low-water mark (Kafka calls its analog the log-start offset) is the oldest offset still retained after snapshotting and segment deletion — the floor below which the log no longer exists.
Bottom row: who obeys which mark. Consumers may read only up to the high-water mark; the leader refuses to serve entries beyond it, so no reader ever sees an uncommitted entry. Snapshot and compaction operate below the low-water mark: once state up to some offset has been captured in a snapshot and no counted replica needs the raw entries, segments below the low-water mark are reclaimed. Between the two marks lies the live, committed, retained region of the log — the entries that are both safe to read and still on disk.
Bottom strip: the operational surface. Watermarks expose the health of the whole replication system. HW lag (leader LEO minus HW) reveals how far ahead the leader is of the commit boundary — a proxy for replica health. LW retention (HW minus LW) reveals how much history is available for catch-up. Truncation-on-election events reveal how much uncommitted data a failover discarded. And monotonicity checks confirm the marks never move backward in ways that would violate read safety. The diagram shows the HW pinned to the minimum replicated offset and the LW as the truncation floor, with consumers gated by one and reclamation gated by the other.
End-to-end flow
Trace a write and its watermarks through a three-replica partition. The leader appends record at offset 1,000 to its log; its LEO is now 1,001. The high-water mark is still 995, because follower B has only replicated through offset 995 while follower A is at 998. The producer that wrote offset 1,000 has not yet been acknowledged (assuming acks require the record to be committed), and no consumer can read it — it is beyond the HW.
Replication proceeds. Follower A fetches through 1,001 (LEO 1,001); follower B fetches through 1,000 (LEO 1,001 after persisting). The leader recomputes the HW as the minimum LEO of the in-sync set — now 1,001 — and advances it. The record at 1,000 is committed: the producer is acknowledged, and consumers reading this partition can now see it. Notice the HW moved only when the slowest counted replica caught up; that is the safety property in action.
Now a failover. The leader crashes with LEO 1,050 but an HW of only 1,001 — it had appended fifty records that were not yet on enough followers. A new leader is elected from the in-sync followers; suppose the winner had replicated through 1,030. Records 1,002 through 1,030 exist on the new leader; records 1,031 through 1,050 were never committed and, on any follower that had them, are now truncated back to the new leader's log — they were beyond the old HW, so no consumer ever saw them and no producer was ever told they succeeded. The high-water mark's promise holds across the election: everything a consumer read (≤ 1,001, then growing) survives; everything uncommitted is safely discarded. The new leader then re-advances the HW as followers re-sync.
Meanwhile the low-water mark is doing its own work. A snapshot of the state machine up to offset 900 completed an hour ago, and all in-sync replicas are well past 900, so the log segments below the low-water mark (say, offset 800) are eligible for deletion; the system reclaims them and advances the LW. A brand-new replica joining the cluster does not replay from offset 0 — it installs the latest snapshot (state as of 900) and then streams live entries from the low-water mark forward, catching up to the HW in minutes instead of hours. If instead the LW had been advanced past 900 while a lagging follower still sat at 850, that follower would find its needed entries gone and be unable to catch up — which is exactly the orphaning failure the low-water mark policy must never cause.