Standing up a second region for a stateless web service is almost boring: containers are small, they start in seconds, and an idle replica costs cents. Doing it for LLM inference is a different problem, because the unit you are replicating is not a container — it is a GPU node holding hundreds of gigabytes of weights resident in high-bandwidth memory, which takes minutes to become useful and bills every second whether or not a request lands on it. Multi-region serving is therefore a capacity and state question long before it is a routing question: the four honest reasons to add a region, what weight distribution and cold starts cost, how headroom math changes when a region must absorb its neighbour, and why a conversation ends up pinned to the region whose KV cache holds it.
Four honest reasons to add a region
There are four defensible motives, and it is worth naming which one you actually have before spending anything. Latency: users far from your single region pay the round trip twice, once to send the prompt and again on the first token. Data residency: a legal requirement that a customer's prompts never leave a jurisdiction. Capacity access: the accelerator you need has quota in another region and not in yours. Blast radius: a regional control-plane or power event takes the service down.
Notice how different these are. Residency and capacity are hard constraints: no amount of tuning inside one region satisfies them. Latency and blast radius are soft, and both can often be improved more cheaply. Teams get into trouble buying a multi-region architecture on the soft reasons, then paying hard-constraint prices.
Latency: check the number before you buy the region
The latency argument transfers badly from ordinary APIs. For a REST endpoint that answers in 20 ms, an added 150 ms of intercontinental round trip is the entire response. For an LLM streaming several hundred tokens over ten or twenty seconds, that same 150 ms is a rounding error on completion time.
Where it does show is time to first token, the metric users read as responsiveness. Round trip lands on TTFT directly, alongside queueing and the prefill pass. If your budget is 300 ms and the wire eats half, geography is a real problem. If TTFT is already 900 ms because prompts are long and the batch is deep, moving GPUs closer buys a fraction nobody perceives, and the same money spent shortening the queue buys more. Measure the split between wire and server time first; only the wire part is recoverable by adding a region.
Data residency: the reason that does not negotiate
Residency is the motive that most often justifies the bill, because there is no engineering substitute. If a contract or regulation says a customer's data is processed inside a jurisdiction, inference for that customer runs on accelerators inside that jurisdiction, full stop.
What surprises people is how much of the stack is in scope. The prompt is obviously user data. So is the completion, the request log, the trace, the evaluation sample forwarded to a central bucket, and any prompt-cache or embedding index derived from the text. The KV cache in GPU memory is user data too. A residency-driven deployment is therefore not one global service with a regional front door; it is a set of largely independent stacks sharing a pipeline and little else. Tenants get a home region at onboarding and stay there, and the hard part becomes aggregating telemetry without carrying payloads out.
Capacity access: sometimes the region picks you
The least discussed driver is the most common in practice: the GPUs are where the GPUs are. Accelerator supply is not evenly spread across a cloud's regions. A new generation lands in a handful of regions first, quota is granted per region, and the topology you need — NVLink inside the node, a non-blocking InfiniBand or high-speed Ethernet fabric between nodes — may exist in only some of them.
So teams end up multi-region not because users demanded it but because the only way to get another few hundred accelerators was to take them elsewhere. That is a different architecture from the latency case: the regions are not serving distinct user populations, they are one logical pool that happens to be split. Work should be placed by available capacity rather than proximity, and the two fleets will differ in generation, node shape and price.
The idle-GPU problem nobody budgets for
Here is the economic fact that makes GPU multi-region unlike everything else you have replicated. A stateless service in a standby region scales to near zero, and when traffic arrives an autoscaler adds pods in seconds. An inference replica cannot: a cold one must acquire a GPU node, pull the weights, load them into device memory and warm its kernels before it serves anything. That is minutes.
So a meaningful standby presence means holding real accelerators warm with weights already resident, and accelerators bill by wall clock, not by tokens. A standby sized to take half your load costs what half your production fleet costs and produces nothing. This is the largest line item in a multi-region inference budget, and the one most often discovered after the design is approved. Every choice below is an attempt to make that idle capacity smaller or less idle.
Distributing weights and the cold-start bill
Weights are the heaviest artefact in the system. A large model in 16-bit precision runs to hundreds of gigabytes; quantised variants shrink that a lot, but not to container-image scale. Those bytes must exist in every region: replicated object storage, cross-region transfer on every release, a per-region copy you store.
Pulling them at start time is the expensive path. Reading hundreds of gigabytes from regional object storage, through the host page cache and out to device memory, is a multi-minute operation — and it is exactly what you are asking for during a capacity emergency. Mitigations point the same way: keep weights out of the container image so image pulls stay fast; pre-stage them onto local NVMe before you need them; use a shared regional cache so the hundredth node does not repeat the first one's download; and where the stack supports it, move bytes from NVMe to GPU memory with GPUDirect Storage instead of bouncing through host RAM. Staging is a scheduled pipeline, not a startup side effect.
Headroom and the N+1 question across regions
Inside one region, capacity planning is familiar: size for peak, add spare replicas so losing one or two does not breach the SLO. Across regions the question changes shape. If region A must absorb region B's traffic, A's spare capacity is not one replica — it is all of B. Two regions each running at 90 percent are not resilient; they are two regions that both fall over the moment either one fails.
Full mutual coverage means each region runs near 50 percent utilisation, doubling the fleet. The honest middle grounds are degradation and time-zone arbitrage. Degradation: decide in advance what you shed when a region is lost — shorter maximum outputs, throttled free tiers, a fallback to a smaller model — so the survivor absorbs a reduced load, not all of it. Time-zone arbitrage: if regional peaks do not overlap, each region's off-peak trough is genuine surge capacity for its neighbour.
Active-active or active-passive, framed as capacity
The choice between both regions serving and one region waiting is usually framed as a failover topic. From a capacity standpoint it is simpler: active-active means your standby capacity is doing paid work; active-passive means it is not.
That accounting favours active-active heavily for expensive accelerators, with a second benefit: a path serving production traffic continuously is a path you know works, whereas a passive region's readiness is a belief until you test it. The price is that every stateful concern — rate limits, quota counters, conversation state, cache warmth — now has two live writers. The pragmatic compromise is a pilot-light fleet: a small always-on footprint in the second region, big enough to keep weights staged and the deploy path exercised, sized to grow rather than to carry full load cold.
KV cache and why a conversation is pinned to a region
Now the state problem. During generation, every token's attention keys and values sit in GPU memory, typically in paged blocks managed by the serving engine. For a long context that is gigabytes per sequence, living in the HBM of one specific replica. It is not something you replicate across an ocean; by the time you shipped it the request would be over.
Multi-turn conversations make this sticky beyond a single request. If a session's prefix is cached in region A, sending its next turn to B does not merely lose an optimisation — B must prefill the entire transcript before emitting a token, which for a long conversation dominates the request. So routing carries session affinity on top of any proximity logic, and the prefix-cache hit rate you measured in one region quietly degrades once the same traffic is split across two.
Failover mid-stream, and when one region is the right answer
Because that state is local and volatile, an in-flight generation cannot be handed to another region. If the replica dies mid-stream its KV cache dies with it, and the tokens already sent are all that survive. Two honest options: fail the stream and let the client retry the original prompt, paying prefill twice; or resume by re-submitting the prompt plus the partial output. Both are re-prefills, the second merely hides the seam. Either way, make retries idempotent and drain a region by refusing new sessions rather than cutting live ones.
Which leaves the quiet part: for many services, one region with real headroom and an honest SLO beats two regions run thin. A second region roughly doubles fleet cost and deploy surface and adds split-state bugs you did not have. If your driver is residency or capacity access, pay it and move on. If it is latency or blast radius, price the alternative — more headroom, faster prefill, an availability target you actually meet — first.