Why architecture matters here
Teams reach for distributed locks for two very different jobs, and the architecture that suffices depends on which one you are doing. The first is efficiency: preventing duplicate work, like two cron workers both rebuilding the same cache. If the lock fails and two workers run, you waste CPU; nothing corrupts. The second is correctness: guarding an invariant, like ensuring only one node applies a schema migration or writes a checkpoint. If the lock fails there, you get split brain, torn state, and data loss. Efficiency locks can be cheap and slightly wrong; correctness locks cannot, and pretending an efficiency-grade lock is a correctness lock is the root cause behind a whole genre of production postmortems.
The reason correctness is hard is that a lock over a network is a claim about time. The service grants a lease for, say, ten seconds; the client believes it has ten seconds of exclusivity. But the client's clock of experienced time and the service's wall clock disagree the moment the client pauses. A 15-second GC pause after reading 'lease valid' leaves the client executing its critical section while the service has already expired the lease and granted it to someone else. No amount of shortening or lengthening the TTL fixes this — it only moves the window. The fix has to live at the resource: writes must carry proof of which grant they belong to, which is precisely what fencing tokens provide.
Architecture also decides availability. A lock service that loses state on restart hands out overlapping grants; one that is a single node makes the lock a single point of failure for every workflow that acquires it. Consensus-replicated lock state — the ZooKeeper/etcd design — is the standard answer because it keeps grants linearizable across service crashes, at the cost of a quorum round-trip per acquire, which is exactly the trade a correctness lock should make.
The architecture: every piece explained
Top row: the actors. Clients A and B race for the lock; the lock service is a small replicated state machine — etcd over Raft, ZooKeeper over ZAB — whose replicated log records every grant, renewal, and release. Because the state machine is deterministic and the log is agreed by quorum, a service-side crash cannot forget who holds the lock or accidentally double-grant it: a new leader replays the same decisions.
Second row: the mechanisms inside the service. The lease manager attaches a TTL to every grant; the holder must renew (keepalive) before expiry or the lock is released on its behalf — this is the liveness bound that stops a crashed holder from blocking the world forever. The session layer is how ZooKeeper expresses the same idea: the lock is an ephemeral node tied to a heartbeating session, and session expiry deletes it. The fencing counter is a monotonic integer bumped on every grant — etcd's ModRevision or ZooKeeper's zxid/czxid serve naturally — returned to the holder alongside the lease. The wait queue orders contenders: in ZooKeeper, sequential ephemeral children where each waiter watches its predecessor (avoiding thundering herds); in etcd, a sorted keyspace per lock with watches on the prior key.
Third row: where correctness is actually enforced. The protected resource — a storage service, a database row, an internal API — stores the highest fencing token it has accepted and rejects any request carrying a lower one. This check is the difference between a lock that is advisory and one that is safe: when Client A pauses, loses the lease, and resumes to write with token 33 after Client B already wrote with token 34, the resource — not the lock service, which A may not have consulted — refuses the stale write. If the resource cannot check tokens (a plain filesystem, a third-party API), you do not have a correctness lock, whatever the lock service promises; the honest fallbacks are CAS on the resource itself or restructuring for idempotency.
End-to-end flow
Acquire. Client A creates a key under the lock's prefix bound to its lease (etcd: a transaction that succeeds only if the lock key is absent; ZooKeeper: create an ephemeral sequential znode and check it is the lowest). On success the service returns the grant plus the fencing token. Client B's create lands behind A's; B sets a watch on A's key and parks. The acquire cost is one quorum write — single-digit milliseconds in-region.
Hold and renew. A's client library heartbeats the lease at a fraction of the TTL — renew every 3s on a 10s lease is a common shape, tolerating two missed renewals before expiry. Every renewal is also a quorum write; this is why lock services cap sustainable lease counts and why one lease per process (with many locks attached) beats one lease per lock. Meanwhile A performs its critical section, attaching token 33 to every mutation it sends to the protected resource, which records 33 as the high-water mark.
Expiry and handoff. A stalls — GC, VM migration, a partition between A and the service. Renewals stop; the lease expires; the service deletes A's key. B's watch fires, B confirms it is now the lowest waiter, and B receives the lock with token 34. B begins writing; the resource's high-water mark advances to 34. When A thaws, two things can happen, both safe: A's next renewal fails with 'lease not found,' telling it to abort; or — the dangerous path — A writes first, before noticing, and the resource compares 33 against 34 and rejects the write. The pause-and-resume hole is closed at the last possible moment, by the component that actually owns the data.
Release. The polite path: A finishes, deletes its key, B is notified immediately — no waiting out the TTL. Releases should be idempotent and guarded by the token too, so a delayed duplicate release from an old holder cannot free the lock under the new one.