Why architecture matters here
LWT misuse is common. Teams use LWT for high-QPS counters and see p99 collapse. Or for rate-limit checks — where an app-level solution scales better. The architecture matters because LWT is a specific tool for linearizable CAS on rare conditional writes.
With the pieces in mind, you can pick LWT correctly and design around its cost.
The architecture: every piece explained
The top strip is the flow. Client CAS issues an INSERT ... IF NOT EXISTS or UPDATE ... IF. Coordinator runs Paxos across replicas. Prepare proposes a ballot; replicas promise not to accept lower ballots. Propose + Commit sends the value; on majority acceptance, coordinator commits.
The middle row is the trade-offs. SERIAL consistency guarantees read-latest for the paxos-committed state. Contention penalty is severe on hot keys — retries thrash. Metrics track contention + latency. Batch caveats: LWT batches are atomic per-partition only.
The lower rows are guidance. Alternatives: app-level idempotency + designed CAS keys can be cheaper for high-QPS. Workload fit: LWT is right for rare, correctness-critical CAS. Ops handles schema design + hotspot detection.
End-to-end flow
End-to-end: a user registration checks unique username via INSERT IF NOT EXISTS. Coordinator runs paxos. If username taken → returns [applied false]. If free → commits. Latency: 40ms typical, 3-4x a normal write. Metrics show 1% contention rate — acceptable. Compare to using LWT for rate-limit counter (which the team correctly did NOT do): under load, LWT contention would cause p99 to spike into seconds.