Why architecture matters here
A distributed lock is architectural because it must provide a safety guarantee — at most one holder acts on the resource — over a system that offers none of the primitives a single-machine mutex relies on. There is no shared memory, so the lock state lives in an external service every client must agree to consult. There is no reliable clock shared across machines, so lease expiry is judged against imperfect local clocks. And there is no way to distinguish a crashed process from a slow one over a network, so the lock cannot assume a holder that has gone quiet is dead. Every design decision in a distributed lock is a response to one of these absences, and skipping any of them produces a lock that works in testing and fails under the exact conditions — crashes, pauses, partitions — it was meant to handle.
The first pillar is that the lock must be a lease, not a claim. If acquiring the lock set a flag that only the holder could clear, a holder that crashed would hold the lock forever and deadlock the whole system. So the lock is granted for a bounded time — a TTL — after which the lock service considers it released regardless of what the holder is doing. This makes crashes self-healing: a dead holder's lease simply lapses and the next waiter proceeds. The cost is that the lease can also expire under a holder that is still alive but slow, which is the crux of the whole problem. The lease length is therefore a fundamental trade-off: too short and healthy holders lose the lock mid-work under normal pauses; too long and a real crash blocks everyone for the full TTL.
The second pillar, and the one most implementations omit, is fencing. A lease bounds how long a crash blocks progress, but it does nothing about a slow holder that revives after expiry and acts on stale belief. The only robust defense is to make the protected resource itself reject stale actors. The lock service issues a fencing token — a number that strictly increases with every grant — and the holder must present its token on every operation against the resource. The resource remembers the highest token it has honored and refuses anything lower. A revived old holder necessarily carries a lower token than whoever acquired the lock after it, so its writes are rejected at the resource. This turns 'at most one holder believes it holds the lock' — which is impossible to guarantee — into 'at most one holder can successfully act', which is achievable and is the guarantee that actually matters.
The third architectural fact is that the lock service itself must be highly available and consistent, which is why it is usually built on a consensus store rather than a single node. If the lock service is a single process, it is a single point of failure whose death takes down everything that depends on locking; if it is replicated without consensus, two replicas can grant the same lock during a partition and mutual exclusion breaks at the source. So production lock services are backed by a quorum-based, consensus-replicated store, so that a majority must agree on who holds a lock and the answer survives node failures and partitions. This is also why 'just use a single Redis key' is a subtly dangerous pattern for locks that guard important state: without consensus and without fencing, it is correct only until the failure it cannot handle occurs. The architecture exists precisely to be correct across those failures, and each piece — consensus store, lease, renewal, fencing token — closes one of the gaps a simpler scheme leaves open.
The architecture: every piece explained
The top row is acquisition. Many clients across the fleet race to hold the lock; by design all but one will fail or wait. The lock service is the arbiter — typically a consensus-backed store or coordinator — that decides who wins and records the current holder durably. On a successful acquire the service grants a lease with a TTL: the client holds the lock only until that time expires, so a crashed holder's grip releases automatically. Alongside the lease the service issues a fencing token, a monotonically increasing number the holder must carry on every action against the protected resource so stale holders can be detected and rejected.
The middle row is holding and using the lock. The holder is the single client that won, now performing the protected work. Because work often outlasts a safe TTL, the holder runs renewal — a periodic heartbeat that extends the lease while it is still making progress, keeping the lock without setting the TTL so long that a real crash blocks others for ages. The protected resource — a database row, a file, an external system — is where fencing is enforced: it checks the presented token against the highest it has seen and refuses stale ones. Meanwhile the waiters are the clients that lost the race; they either poll with backoff or, better, watch for a release notification so they can acquire promptly when the lock frees.
The third row holds the two safety mechanisms that handle the hard cases. The expiry path is the crash-recovery route: if the holder stops renewing — because it died, hung, or was partitioned away — the lease lapses, the lock service frees the lock, and a waiter acquires it, all without the dead holder's cooperation. The split-brain guard is fencing doing its job: if the original holder was only slow and revives after expiry, the new holder already acquired the lock with a higher token, so the resource rejects the revived holder's lower-token writes. These two together are what make the lock safe under the very conditions — crash versus slowness, indistinguishable over a network — that break naive locks.
The ops strip names what operators must watch. Lease duration versus actual work time is the central tuning relationship: leases must be long enough to survive normal pauses but short enough that a crash does not block everyone for too long. Clock skew matters because leases are judged against clocks, and badly skewed clocks can make a lease look expired or valid when it is not. Renewal failures are the early warning that a holder is losing its grip. And lock contention — how many clients pile up waiting — tells you whether the lock has become a serialization bottleneck that is throttling the whole system.
End-to-end flow
Walk the happy path. A client wants to run a job that must not run concurrently, so it asks the lock service to acquire the lock with a TTL of, say, thirty seconds. The consensus-backed service records this client as the holder, returns success, and hands back a fencing token — a number one higher than the last grant. The client begins the work. Because the job takes two minutes, longer than the TTL, the client runs a renewal heartbeat every ten seconds, each renewal extending the lease another thirty seconds so the lock never lapses while work is ongoing. Every write the job makes to the protected resource carries the fencing token, and the resource, seeing a token equal to or higher than its recorded maximum, accepts them. When the job finishes, the client releases the lock explicitly, and the next waiter acquires it immediately.
Now the crash path. Halfway through, the holder's process dies — a kill, an out-of-memory, a machine reboot. It stops sending renewals. Thirty seconds after the last renewal, the lease lapses; the lock service, seeing no renewal, considers the lock free. A waiter that was polling or watching acquires the lock, receives a fencing token higher than the dead holder's, and proceeds. The system self-healed: no operator intervention, no manually cleared stuck lock, just a bounded delay equal to the remaining lease. This is exactly why the lock is a lease and not a claim — the TTL is what turns a fatal deadlock into a short pause.
Now the dangerous path, the one fencing exists for. The original holder did not crash; it froze — a long garbage-collection pause, a stalled disk, a network partition — for forty seconds, longer than its lease. During the freeze its lease lapsed and a second client acquired the lock with a higher fencing token and began its own work on the resource. Then the first holder unfreezes, still believing it holds the lock, and tries to write to the resource carrying its now-stale, lower token. Without fencing, this write would land and two holders would have corrupted the resource together. With fencing, the resource has already seen the second holder's higher token, so it rejects the first holder's lower one outright. The first holder's write fails, it learns it no longer holds the lock, and mutual exclusion held even though, for a moment, two processes each believed they were the holder. This is the single most important scenario in distributed locking, and the reason a lock without fencing is not safe for guarding important state.
Finally, the contention and clock cases that shape day-to-day operation. If dozens of clients contend for one lock, the lock serializes them and becomes a throughput ceiling; the architectural response is to shard the lock — lock per key or per partition rather than one global lock — so independent work does not queue behind unrelated work. And because lease expiry is judged against clocks, significant clock skew between the lock service and clients can make a lease appear expired early or valid late, so operators keep clocks synchronized and, critically, do not rely on wall-clock comparisons for safety — fencing tokens, which are monotonic regardless of clocks, are what actually guarantee correctness. The lease TTL is a liveness mechanism tuned against clocks and pauses; the fencing token is the safety mechanism that does not depend on any clock at all, and keeping those two roles distinct is the heart of running distributed locks well.