Why architecture matters here
The economics force the architecture. HBM is the most expensive memory in the datacenter per gigabyte, and a serving GPU has a fixed, small amount of it — after weights and activation scratch, maybe 40–60GB is left for KV. Divide that by the per-token KV footprint (which grows with layers, heads, and head dimension) and you get a hard ceiling on concurrent context tokens. Hit the ceiling and you have exactly three choices: reject requests, preempt and recompute someone's cache from scratch, or move cache somewhere else. The first caps throughput, the second wastes the compute you already paid for, and only the third — offloading — lets you serve more concurrent sequences than HBM alone could ever hold. On a serving box, host RAM is often 10–20x the size of HBM and NVMe is another order of magnitude beyond that, so the tiers below the GPU are vast by comparison.
The catch is that those tiers are also slower, and attention is latency-critical. Decode is memory-bound: each generated token reads the entire KV cache for its sequence. If a needed block is sitting on NVMe when the GPU wants it, the decode step stalls until it arrives. So the architecture is not merely 'put cache elsewhere' — it is a bet that you can predict which blocks will be needed soon enough to prefetch them, and reuse enough shared prefix cache to avoid moving anything at all. Get the prediction right and offloading is nearly free capacity; get it wrong and every token pays a PCIe round trip.
This is why the design lives or dies on locality and scheduling, not on raw tier size. A request that is actively decoding must have its blocks resident; a request that is queued or paused can be spilled; a system prompt shared by a thousand tenants should be computed once and reused forever. The architecture matters because it turns a blunt capacity wall into a smooth, tunable trade between memory cost and tail latency — and because the wrong version of it degrades so sharply that teams often conclude offloading 'doesn't work' when in fact their eviction policy was fighting their access pattern.
The architecture: every piece explained
Top row: the resident tier and its bookkeeping. GPU HBM holds the blocks for sequences actively decoding right now. The block manager is the heart of the system: KV is stored in fixed-size blocks (say 16 tokens per block per layer), each block is a page, and a per-sequence block table maps logical token positions to physical block ids — this is paged attention, and it is what makes non-contiguous placement and offloading possible in the first place. The manager keeps reference counts so shared blocks (copy-on-write forks, shared prefixes) are freed only when the last sequence releases them. The eviction policy decides which resident blocks to demote when HBM is full — typically LRU, but augmented with reuse hints: never evict a block that a decoding sequence will read this step; prefer evicting queued or paused sequences. The prefix cache indexes blocks by a hash of their token content so an identical prompt prefix maps to already-computed KV and skips prefill entirely.
Middle row: the lower tiers and the machinery that moves data. The host RAM tier is a pool of pinned (page-locked) host memory used as staging — pinned so the GPU's DMA engines can copy to and from it without the OS paging it out mid-transfer. The NVMe/SSD tier holds cold blocks that overflow host RAM: paused conversations, large prefix caches, long-tail sessions. Copy engines are the GPU's dedicated DMA units that move blocks asynchronously, overlapping transfer with compute so a prefetch does not block the decode kernel. The remote KV store extends the hierarchy across the network: a shared, disaggregated cache (over RDMA or a fast object tier) that lets a prefill on one node hand its KV to a decode on another, or lets many replicas share one warm prefix cache.
Bottom rows: the control plane. The scheduler admits requests and drives eviction against a tier budget: it knows each running sequence's block footprint, decides who is resident, who spills, and who waits, and it issues prefetches ahead of need. Layout and quantization keep the moved bytes cheap: blocks are laid out contiguously per layer for coalesced transfer, and KV is often stored at FP8 or INT8 rather than FP16, halving or quartering both the memory footprint and the transfer time at a small, measurable quality cost. The ops strip names the four numbers that matter: cache hit rate, transfer bandwidth, decode stall time attributable to paging, and occupancy per tier.
End-to-end flow
Follow a burst of chat requests through an offloaded server. A new request arrives sharing a 2,000-token system prompt with thousands of prior sessions. The scheduler hashes the prompt prefix, hits the prefix cache, and finds the KV for those 2,000 tokens already resident (or stageable from host RAM). Prefill therefore only processes the user's 40 new tokens instead of 2,040 — a 50x reduction in prefill compute for that request — and the request enters decode almost immediately with its shared prefix blocks reference-counted, not copied.
Decode proceeds token by token. The block manager keeps this sequence's active blocks in HBM. As more concurrent requests admit, HBM fills; the eviction policy looks for victims. Two of the sessions have gone idle — the users are reading, not typing — so their KV blocks are demoted: the copy engine asynchronously DMAs them to the pinned host RAM tier and frees the HBM. Because the transfer overlaps with other sequences' decode kernels, no active request stalls. The block table for the paused sessions now points at host-resident blocks marked non-resident.
One paused user sends a new message. The scheduler must bring that sequence's KV back before it can decode. It issues a prefetch: the copy engine streams the blocks from host RAM to HBM while the token is still being tokenized and queued, so by the time the decode step for that sequence runs, its blocks are resident. If the blocks had been spilled all the way to NVMe, the prefetch would be slower, and the scheduler would have queued the request one or two steps to hide the latency — a small, bounded delay rather than a hard rejection. Under heavy long-context load, a prefill on a dedicated prefill node computes a large KV and ships it over the remote KV store to a decode node, which pages it in — the two halves of the request never share a GPU.
Now the stress case: a flash crowd of long-context requests arrives and everyone is actively decoding, so nothing is safely evictable. HBM pressure forces the scheduler to spill blocks belonging to running sequences, and now some decode steps must page their blocks back in every step — the dreaded thrash. The scheduler detects rising stall time and switches strategy: it reduces the running batch size, letting fewer sequences decode with fully resident caches while the rest wait in the queue with their KV parked on a lower tier. Throughput dips but tail latency stays bounded, and the system sheds the overload gracefully instead of collapsing into a paging storm.