Read-replica routing architecture — consistency tokens, lag awareness, and read-your-writes at scale

Why architecture matters here

The consistency model you actually need is narrower than it first appears, and getting the scope right is what makes the design tractable. Almost nobody needs full linearizability from their read path. What users notice — and what generates the bug reports — is the violation of two much weaker properties. Read-your-writes: after I change something, I see my change. Monotonic reads: I never see the world move backwards. A user who does not see someone else's update for 500ms will never notice; a user who does not see their own update immediately files a bug within seconds. That asymmetry is the entire design opportunity, because per-session guarantees are dramatically cheaper than global ones.

Monotonic reads is the sneakier of the two, and it is the one that pure round-robin routing breaks even for read-only workloads. Suppose replica A is 20ms behind and replica B is 900ms behind. A user loads a page from A and sees a comment. They refresh; the round-robin lands them on B, which has not received that comment yet. The comment disappears. They refresh again, land on A, and it returns. Nothing is broken — every read was internally consistent and every replica was healthy — but the user watched time run backwards. Any routing policy that can move a session between replicas of differing freshness has this bug latent inside it, and it surfaces as 'the site is flaky' rather than as anything a query log would flag.

The reason this cannot be fixed by 'just making replication fast' is that lag is not a constant you can engineer away; it is a random variable with a very long tail, and the tail is correlated with exactly the moments you care about. Replication lag sits near zero for hours, then a bulk update, a long-running transaction on the primary, a vacuum, or a replica checkpoint stall pushes it to seconds or minutes. The distribution is not merely wide — it is bimodal and its bad mode arrives during peak load, when the primary is busiest and the replicas are furthest behind. A design whose correctness assumes 'lag is usually small' is a design that is correct exactly when nothing interesting is happening.

So the architecture must treat freshness as a runtime property to be checked, not a deployment property to be assumed. Every read either does not care about freshness (an analytics rollup, a cold cache fill) or has a specific causal requirement (this session wrote at LSN X and must not read anything older). The router's job is to know which, cheaply, on every request, and to have a defensible answer when no replica qualifies. Framed that way, replica routing stops being a load-balancing problem and becomes a small distributed-systems problem — which is why it deserves an explicit component rather than a connection-string change.

The architecture: every piece explained

Read classification (top-left). Before anything can be routed, each read must be labeled with the freshness it requires, and the labeling must be explicit. Three classes cover practically everything. Strong reads must see all committed writes and go to the primary — the balance check before a transfer, the row you are about to update. Causal reads must see this session's own writes but tolerate lag on everyone else's — the profile page after a save, which is the overwhelming majority of user-facing reads. Relaxed reads tolerate arbitrary staleness — dashboards, exports, recommendation features. The failure mode to design against is defaulting to relaxed: a system where unlabeled reads silently get the weakest guarantee will accumulate stale-read bugs faster than anyone can find them. Default to causal, make relaxed opt-in, and the worst outcome of forgetting a label is a little extra primary load rather than a correctness bug.

The consistency token and session store (top-right). This is the heart of the design. Every write returns a position in the replication stream: Postgres gives you pg_current_wal_lsn(), MySQL an executed GTID set. The router stores that token as the session's write watermark — the point in history the session has personally caused. Where it lives determines what the design can do: in-process is free but dies with the pod, a signed cookie or request header travels with the user across instances, and a shared Redis keyed by session works across services at the cost of a network hop. Note the watermark only needs updating on writes and can be discarded after enough time has passed that every live replica must have caught up, so the storage cost is bounded and small.

Lag measurement and the replica pool (middle row). The router keeps a live view of each replica's replay position, and the unit matters more than teams expect. Byte lag (the WAL/binlog distance between primary and replica) is cheap to read and is the right thing to compare against a token, because it answers exactly 'has this replica replayed LSN X?'. Time lag (how old the last replayed transaction is) is what SLOs are written in, but it is treacherous: on an idle database it reads as large simply because no new transactions exist to replay, so a naive monitor ejects every healthy replica at 3am on a quiet Sunday. Route on byte lag, alert on time lag, and never wire time lag into an ejection rule without an idleness guard.

Fallback policy and ejection (lower row). Two independent controls sit under the pool. The lag monitor ejects replicas that exceed a threshold, removing them from the pool entirely so they neither serve nor are considered — this is the coarse, per-replica, health-style control. The fallback policy handles the per-request case where a session's watermark disqualifies every remaining replica, and it has exactly three options: send the read to the primary (costs primary load, always correct, the right default), wait briefly for a replica to catch up (adds latency, works when lag is milliseconds), or serve stale and mark the response (only defensible when the caller genuinely opted into relaxed). The policy must be explicit, because the implicit version — whatever the code happens to do when the eligible list comes back empty — is usually 'serve stale silently', the bug this architecture exists to prevent.

End-to-end flow

A write. A user saves their profile. The router classifies the statement as a write and sends it to the primary — no cleverness here, all writes always go to the primary. The transaction commits at LSN 0/16B3A28. Before returning, the router captures that LSN and stores it as the session's write watermark, keyed by session id in Redis with a 60-second TTL. The response goes back to the user. Total added cost: one pg_current_wal_lsn() call and one Redis SET, both sub-millisecond, on the write path only — which is the cheap path precisely because writes are the minority.

The read that follows. The page reloads and issues SELECT * FROM profiles WHERE user_id = ?. The router classifies it as a causal read and fetches the session's watermark: 0/16B3A28. It walks the replica pool comparing replay positions. Replica A is at 0/16B3A28 or beyond — eligible. Replica B is at 0/16B3990, which is behind the watermark — not eligible for this session, though it remains perfectly fine for any session that has not written recently. Replica C was ejected by the lag monitor an hour ago. The read goes to A, returns the new name, and the user sees their save. The bug that started this article never happens, and it never happens by construction rather than by timing luck.

The read from a session that did not write. A different user loads the same profile page. Their session has no watermark at all — they have written nothing, so nothing about their causal history constrains where they can read. The router skips the comparison entirely and routes to any healthy replica by least-connections, landing on B. B is 900ms stale, so this user might see a version of the profile from a second ago. They cannot tell, and no guarantee has been violated: read-your-writes is vacuous for a session with no writes. This is where the load reduction actually comes from — the large majority of sessions in a read-heavy system are in exactly this state, and they route freely.

The degraded case. A bulk migration runs on the primary and every replica falls two seconds behind. Our profile-saving user reads again. The router compares the watermark and finds no eligible replica: A, B, and C are all behind the user's own write. The fallback policy fires and sends the read to the primary. The user sees correct data; the cost is primary load for the duration of the lag spike, and the replica_fallback_total counter climbs — which is exactly the signal that should page someone, because it is the leading indicator that the read-scaling benefit has evaporated. The system degrades toward 'expensive but correct' rather than 'cheap but wrong', and that is the single most important property of the whole design.

Failure modes and mitigations

• Silent stale reads. The catastrophic mode: no eligible replica exists and the router serves from a stale one anyway because the empty-list case was never handled. There is no error, no metric, no log — just occasional wrong answers that nobody can reproduce. Mitigation: make the empty-eligible-list branch explicit and loud. Default it to primary fallback, count every occurrence, and treat 'serve stale anyway' as a decision that requires an opt-in flag on the call site.
• Time lag on an idle database. A time-lag-based ejection rule sees no transactions replayed for 30 seconds at 3am, concludes every replica is 30 seconds stale, and ejects the entire pool — sending all traffic to the primary at the moment nothing was wrong. Mitigation: route on byte lag, which is idleness-immune. If you must alert on time lag, suppress it when the primary's LSN has not advanced, or emit a periodic heartbeat write so the metric stays meaningful.
• Watermark lost across instances. The write lands on pod 1, which stores the watermark in local memory; the follow-up read is load-balanced to pod 2, which has no watermark, assumes the session is write-free, and routes to a stale replica. The bug appears only under multi-instance deployment, so it survives every local test. Mitigation: store the watermark somewhere the whole fleet can see — a shared cache, or carried with the request in a signed cookie or header — and test with at least two instances behind a load balancer.
• Failover leaves a watermark from a dead timeline. The new primary's LSN sequence does not line up with the old one, so sessions holding pre-failover watermarks compare against a meaningless position — either disqualifying every replica forever or accepting a stale one. Mitigation: include a timeline or epoch id alongside the token and invalidate watermarks whose epoch does not match the current primary.
• Fallback stampede. Lag spikes, every session's watermark disqualifies every replica simultaneously, and 100% of reads fail over to the primary at once — which was sized for 20% of reads. The primary saturates, which slows replication further, which deepens lag. Mitigation: bound the fallback path with its own concurrency limit or load shedder, and alert on fallback rate early enough to intervene before the feedback loop closes.
• Read-modify-write through a replica. Code reads a row from a replica, computes a new value from it, and writes the result to the primary — silently basing a write on stale state and clobbering an intervening update. This is the most damaging variant because it does not merely display stale data, it persists a decision made from it. Mitigation: any read whose result feeds a write must be classified strong; reads inside a transaction that will write should route to the primary by default, not by developer discipline.

Operational playbook

• Instrument the replica-hit ratio first. The single number that says whether the design is earning its complexity is the fraction of reads served by a replica. If it is 95%, you are getting the scaling benefit; if it has quietly fallen to 40% because lag rose, you are paying full complexity for partial benefit and nobody noticed. Chart it next to fallback rate and treat a sustained drop as an incident, not a curiosity.
• Alert on fallback rate and lag percentiles, not lag averages. Average lag is near zero even when the p99 is four seconds, and the p99 is what breaks users. Chart a lag histogram per replica, alert on p99 crossing the threshold that governs eligibility, and page on fallback rate — the metric that means 'read scaling has stopped working' — rather than on lag itself.
• Make read classification reviewable in code. The routing decision should be visible at the call site — an explicit readPreference: causal | strong | relaxed argument, not a thread-local set three frames up. If a reviewer cannot see which consistency a query gets by reading the query's own code, the classification will drift within a quarter and the drift will be invisible.
• Default to causal, opt into relaxed. Reverse the usual default. A forgotten label should cost you a little primary load, which shows up on a graph and gets fixed, rather than a stale read, which shows up as an unreproducible bug six months later. This single choice prevents more incidents than any amount of lag tuning.
• Rehearse a lag spike in staging. Deliberately stall a replica and confirm the router ejects it, fallback fires, the primary's load limiter holds, and no read returns stale data. This test validates the whole architecture end to end and is almost never written.

Anti-patterns to avoid

• Routing by SQL prefix. Sending everything starting with SELECT to a replica is the classic trap. It routes SELECT ... FOR UPDATE to a read-only replica, misses writes hidden inside stored procedures and CTEs, and ignores session state entirely. Route on an explicit application-level intent, never on string inspection.
• Replicas as a substitute for indexing. If the primary is CPU-saturated by unindexed queries, replicas multiply the same bad plan across more machines and buy you a few months before the same wall. Fix the query first; scale reads because your read volume genuinely exceeds one machine, not because a query is slow.
• Sticky sessions instead of watermarks. Pinning a session to one replica gives monotonic reads but not read-your-writes — the write went to the primary and your pinned replica may still be behind it — and it breaks the moment that replica is ejected. It looks like a simpler version of the watermark design but solves a strictly weaker problem.
• Ignoring the read-modify-write path. Bears repeating as its own anti-pattern: the difference between a stale read displayed and a stale read persisted is the difference between a cosmetic bug and silent data corruption. Treat reads that inform writes as strong, structurally.
• Global lag threshold for all queries. One 'replicas must be under 1s' rule ignores that the analytics dashboard would happily accept a minute while the profile page needs the session's own write. The requirement belongs to the query, not to the cluster; a single global threshold is simultaneously too strict for some reads and too loose for others.