Why architecture matters here
Fencing tokens matter because they close a fundamental correctness gap in distributed locking -- one that's easy to miss and can silently corrupt data, and that leases alone cannot close. Distributed locks are widely used (ensuring exclusive access to a shared resource), and there's a natural but wrong assumption that a lock (with a lease/timeout, for liveness -- so a dead holder's lock is eventually released) is sufficient for exclusive access. But it's not: a process can pause (GC, network, VM suspension) for longer than its lease, during which the lock is granted to another process, and then resume and act -- violating the exclusivity (two processes acting). This gap is easy to miss (the pause scenario is subtle -- it seems like the lease handles dead holders, but a paused-then-resumed holder isn't dead) and dangerous (two processes accessing the resource -- data corruption, silently). Fencing tokens close this gap (the resource rejecting the stale holder's requests) -- making the lock actually safe (exclusive access truly guaranteed, even against paused holders). For anyone using distributed locks to protect a shared resource (a common pattern), understanding fencing tokens (the gap they close, why leases alone don't) is essential to correct locking, and ignorance of it (relying on locks/leases alone) is a subtle, dangerous correctness bug.
The leases-are-insufficient insight is the crucial understanding, and it's why fencing is necessary. A lease (a lock with a timeout) provides liveness (a dead holder's lock is eventually released -- the lease expiring -- so the system isn't stuck if a holder dies). But a lease does NOT provide safety against a paused holder: the problem is that the holder can't reliably know its lease has expired (during a pause, the holder isn't running -- it doesn't see the lease expire; when it resumes, it doesn't know time has passed and its lease expired -- it still believes it holds the lock). And you can't fix this with more careful lease-checking by the holder (the holder might check its lease is valid, then pause, then act -- the pause between the check and the action, during which the lease expires -- a time-of-check-to-time-of-use gap that no amount of holder-side checking closes, because a pause can always occur between the check and the use). So the safety can't come from the holder (which can be paused at any point) -- it must come from the resource (which is always the final arbiter of access). This is why leases alone are insufficient (they provide liveness but not safety against pauses -- the holder can't reliably self-enforce) and why fencing (resource-side enforcement) is necessary. Understanding that leases provide liveness but not pause-safety (the holder can't reliably know its lease expired, so safety must be enforced at the resource) is understanding why fencing tokens are necessary.
And the resource-must-enforce insight is the crucial mechanism detail, and it's where fencing is often gotten wrong. The fencing token mechanism has two parts: the lock service issues monotonic tokens (each grant a higher token), AND the resource checks the tokens (tracking the highest seen, rejecting lower ones). The second part -- the resource enforcement -- is the crucial one, and it's easy to miss. Issuing tokens (the lock service's part) is useless without the resource checking them (rejecting stale ones) -- because the fencing happens at the resource (the resource rejecting the stale holder's request -- based on the token). If the resource doesn't check the token (just accepts any request), the fencing does nothing (the stale holder's request accepted -- the corruption not prevented). So the resource MUST support and check fencing tokens (rejecting requests with stale tokens) -- the enforcement is at the resource, not the lock service. This is a crucial (and often-missed) point: fencing requires the resource to cooperate (checking tokens) -- so fencing works only with resources that support token checking (or a proxy/wrapper that enforces it). If the resource can't check tokens (a resource that just accepts any request), fencing can't be enforced there (a fundamental limit). Understanding that the resource must enforce the token (reject stale tokens -- the fencing happening at the resource, requiring the resource's cooperation) is understanding the crucial mechanism detail and a key limit of fencing.
The architecture: every piece explained
Top row: the problem and the mechanism. The problem: a lock holder pauses (GC, network, VM suspension) longer than its lease -- the lease expires during the pause, and the lock is granted to another process. Stale lock holder: the paused process resumes and acts on the resource (still believing it holds the lock -- not knowing its lease expired) -- while another process also holds the lock -- two processes accessing the resource (the exclusivity violated). Fencing token: the lock service issues a monotonically increasing number per grant (each grant a higher token) -- the token identifying the grant's recency. Resource checks token: the resource tracks the highest token it has seen and rejects any request with a lower (stale) token -- fencing out the stale holder (whose token is now lower than the newer holder's).
Middle row: the details. GC pause / network delay: the causes of the pause -- a long GC pause (the JVM stopping the process), a network delay (the process's messages delayed), a VM suspension (the VM paused) -- any of which can pause the holder longer than its lease. Monotonic counter: the lock service maintains a monotonic counter (incremented per grant -- so each grant's token is strictly higher) -- ensuring the tokens are monotonically increasing (a later grant always has a higher token). Resource enforcement: the crucial part -- the resource checks the token (tracking the highest seen, rejecting lower ones) -- the fencing enforcement at the resource. vs lease alone: why leases are insufficient -- a lease provides liveness (a dead holder's lock released) but not safety against a paused holder (which can't reliably know its lease expired) -- so fencing (resource enforcement) is needed for safety.
Bottom rows: usage and limits. Where used: fencing tokens are used with lock services -- ZooKeeper (the zxid or a sequential node as the token), etcd (the revision or a lease-associated token), and storage systems (that support conditional writes / token checking) -- the lock service providing the monotonic token, the resource checking it. Limits and assumptions: fencing requires the resource to cooperate (check the token) -- so it works only with resources that support token checking (or a proxy enforcing it); a resource that can't check tokens can't enforce fencing (a fundamental limit). The ops strip: token propagation (propagating the token from the lock service through the process to the resource -- the token must accompany every resource request), resource support (ensuring the resource supports token checking -- the crucial requirement; using a resource or proxy that can reject stale tokens), and testing (testing the fencing -- especially the pause scenario, which is hard to trigger naturally -- e.g., fault injection simulating pauses -- to verify the fencing actually works).
End-to-end flow
Trace the fencing token preventing corruption. A process (client 1) acquires a lock to write to a storage resource, receiving fencing token 33 (from the lock service's monotonic counter). Client 1 then pauses (a long GC pause -- longer than its lease). The lock service, seeing client 1's lease expire, grants the lock to client 2, issuing token 34 (higher -- the monotonic counter incremented). Client 2 writes to the resource with token 34 -- the resource, seeing token 34 (higher than any it has seen), accepts it and records 34 as the highest token seen. Now client 1 resumes (its pause ends), still believing it holds the lock (not knowing its lease expired during the pause), and writes to the resource with its old token 33. The resource checks the token: 33 is LOWER than the highest it has seen (34) -- so the resource REJECTS client 1's write (a stale token) -- fencing out client 1. So client 1's stale write is rejected (the corruption prevented -- client 1 fenced out despite believing it holds the lock), and only client 2 (the current holder, token 34) can write. The fencing token (client 1's stale token 33 rejected by the resource) prevented the two-writers corruption that the lock/lease alone couldn't -- the resource-side token check being the enforcement.
The lease-insufficiency and resource-enforcement vignettes show the key insights. A lease-insufficiency case: the team initially relied on a lock with a lease (assuming it guaranteed exclusive access) -- but hit a data corruption (two processes wrote simultaneously) traced to a GC pause (a process paused past its lease, resumed, and wrote while another held the lock) -- realizing the lease provided liveness (releasing the dead-appearing holder's lock) but not safety against the paused holder (which resumed and acted). The fix: fencing tokens (the resource rejecting the stale holder's writes) -- closing the gap the lease couldn't. A resource-enforcement case: the team adds fencing tokens but initially only has the lock service issue tokens (without the resource checking them) -- which does nothing (the stale holder's writes still accepted -- the corruption not prevented). They realize the resource must check the tokens (reject stale ones) -- so they ensure the resource (or a proxy) enforces the token check -- the fencing then working (the enforcement at the resource being the crucial part).
The usage and testing vignettes complete it. A usage case: the team uses ZooKeeper for locking (a sequential znode providing the monotonic token -- the znode's sequence number as the fencing token) and ensures their storage resource checks the token (rejecting writes with a token lower than the highest seen) -- the ZooKeeper-provided monotonic token, checked at the storage resource, providing the fencing. A testing case: the team tests the fencing by injecting pauses (simulating a GC pause -- pausing a process past its lease, then resuming it and having it attempt a write -- verifying the resource rejects the stale write) -- testing the pause scenario (hard to trigger naturally) via fault injection, verifying the fencing actually works. The consolidated discipline the team documents: recognize that a lock/lease alone is insufficient (a paused holder can act after its lease expires -- violating exclusivity), use fencing tokens (the lock service issuing monotonic tokens, the resource rejecting stale ones -- fencing out paused holders), ensure the resource enforces the token check (the crucial part -- the enforcement at the resource, not just issuing tokens), understand the limit (fencing requires the resource to cooperate -- a resource that can't check tokens can't enforce fencing), propagate the token (from the lock service through the process to the resource -- with every request), and test the fencing (especially the pause scenario, via fault injection) -- because fencing tokens close a fundamental correctness gap in distributed locking (a paused holder acting after its lease expires), which leases alone cannot close (they provide liveness but not pause-safety), by having the resource reject stale tokens (the enforcement at the resource being the essential mechanism).