Why architecture matters here
Quantify the interference first. A 8k-token prefill on a modern 70B deployment occupies the GPU for hundreds of milliseconds; every request mid-generation on that GPU emits no tokens while it runs. In continuous batching, one arriving long prompt degrades p99 inter-token latency for dozens of streams — the classic hockey-stick in ITL graphs that appears exactly when marketing sends the long-context announcement. Chunked prefill (slicing the prompt into scheduler-friendly pieces interleaved with decode steps) softens the spike but taxes prefill efficiency and still shares the same HBM bandwidth between phases; it is a truce, not a separation.
The deeper issue is that one pool cannot be provisioned correctly for two jobs. Prefill throughput wants maximum FLOPs and benefits from large sequence-parallel batches; decode throughput wants maximum HBM bandwidth and enormous request-count batches whose weights-read cost amortizes across streams. The optimal batch shape, the optimal parallelism (tensor-parallel degree can differ per phase), even the optimal GPU SKU diverge. A co-located fleet is perpetually mis-provisioned for one phase or the other, and autoscaling cannot fix a ratio that exists inside each GPU.
Disaggregation converts that entanglement into an explicit, tunable system parameter: the prefill:decode pool ratio. Goodput-oriented schedulers (DistServe’s framing) provision each pool against its own SLO and route around the interference entirely. The price is a new data-plane dependency — moving multi-gigabyte KV caches between pools fast enough that the handoff does not itself become the latency spike — and that transfer fabric is where disaggregated architectures are won or lost.
The right objective function makes the case quantitative. Raw tokens-per-second rewards exactly the wrong thing — a co-located fleet can post excellent aggregate throughput while violating both latency SLOs, because prefill and decode each degrade the other’s tail. The metric that matters is goodput: requests per GPU-second that meet both the TTFT and ITL targets. DistServe’s original evaluation reported multi-x goodput gains (and order-of-magnitude tighter latency compliance) from disaggregation at equal hardware, precisely because co-location forces one shared batching policy to split the difference between two workloads that want opposite things. Framed as goodput, the pool split stops being an exotic optimization and becomes the obvious move: it is the only architecture in which each phase’s scheduler can be tuned for its own SLO without a zero-sum fight over the same SMs and HBM.
The architecture: every piece explained
The router. Every request lands at a scheduler that knows both pools’ state. It dispatches the prompt to a prefill worker chosen by queue depth and prefix-cache affinity, and simultaneously reserves KV block capacity on a decode worker so the handoff target exists before prefill finishes. Good routers are SLO-aware: they will queue prefill (hurting TTFT) before they will oversubscribe decode (hurting ITL for everyone already streaming), or vice versa per product policy.
Prefill workers. Optimized for FLOPs: large chunked batches of prompt tokens, often higher tensor-parallel degree, no persistent per-request state after handoff. Their output is the final KV cache for every layer plus the first generated token. Because they hold requests only briefly, their utilization is bursty and queue-depth-driven.
The KV transfer fabric. The load-bearing wall. A 8k-token context on a 70B model is on the order of gigabytes of KV; moved naively after prefill completes, it adds hundreds of milliseconds of dead time. Production systems stream layer-wise: as prefill finishes layer L, layer L’s KV blocks begin transferring while layer L+1 computes, hiding most transfer behind compute. Transport is NVLink within a node, GPUDirect RDMA over InfiniBand/RoCE across nodes, abstracted by libraries like NIXL (Dynamo) or Mooncake’s transfer engine, with paged KV blocks as the transfer unit.
Decode workers. Optimized for bandwidth: hundreds of concurrent streams per GPU, paged-attention KV management, continuous batching of single-token steps. They receive KV blocks into pre-reserved pages and enter the generation loop without recomputation. Prefix cache: a shared (often node-local plus cluster-tier, as in Mooncake) store of KV for common prompt prefixes — system prompts, few-shot templates — letting prefill skip recomputation and sometimes letting decode start with zero prefill at all.
Two refinements matter at the design stage. KV footprint reduction compounds with everything: grouped-query attention, KV quantization (fp8 or int8 cache), and multi-latent attention shrink the per-token cache, which cuts transfer bytes, decode memory pressure, and prefix-cache storage simultaneously — a smaller cache makes the whole disaggregated design cheaper at once. Elastic role assignment softens the pool boundary: rather than two fixed fleets, schedulers like Dynamo’s can flip a worker between prefill and decode roles in seconds, and some deployments run conditional disaggregation — co-located serving at quiet hours, splitting only when load makes interference measurable. The architecture is best understood as a spectrum from fully co-located to fully split, with the operating point chosen by traffic, not ideology. Hardware can specialize too: prefill pools favor FLOPs-dense SKUs while decode pools favor HBM capacity and bandwidth, and heterogeneous fleets exploit exactly that split.
End-to-end flow
Trace a 6k-token RAG prompt end to end. The router hashes the prompt’s prefix blocks and finds the 1k-token system prompt already in the prefix cache on prefill worker P3 — affinity routing sends the request there, and only 5k tokens need computing. Simultaneously the router reserves KV pages for 6k tokens plus generation headroom on decode worker D7, picked for lowest memory pressure. P3 runs chunked prefill; as each transformer layer completes its final chunk, that layer’s KV blocks stream over RDMA to D7’s reserved pages — by the time the last layer finishes, most of the cache has already landed. P3 emits the first token (TTFT stops ticking), forwards the sampling state, and frees its local copy; the request now costs P3 nothing.
D7 splices the request into its continuous batch — one more row in the next step’s batched attention over paged KV. Each iteration reads weights once for the whole batch and each stream’s KV pages, appends one token per stream, and yields tokens to the streaming API. The new request’s arrival changes nobody’s ITL measurably, because no prefill ever runs here — that is the entire point. Generation ends at a stop token; D7 frees the pages (or demotes reusable prefix blocks to the cache tier), and the router’s accounting updates.
Zoom out to the fleet view. TTFT is governed by prefill queue depth plus transfer overlap efficiency; ITL by decode batch size and HBM bandwidth; and the autoscaler watches both to adjust the pool ratio — a burst of long-document summarization shifts capacity toward prefill, an agentic workload of short prompts with long generations shifts it toward decode. Some stacks flex at finer grain by flipping individual workers between roles as the mix drifts.
Multi-turn conversations add a wrinkle worth tracing. When the user’s second message arrives, the conversation’s existing KV may still sit on D7 from the previous turn; a session-affine router sends the follow-up prefill of only the new tokens to a prefill worker, which streams just the incremental blocks to D7 — or, if D7 evicted the session under pressure, the prefix cache tier restores what it can and only the remainder recomputes. This is where KV-centric designs like Mooncake earn their name: the cache is treated as a first-class distributed object with its own placement, replication, and eviction policy, and prefill becomes, in effect, a cache-miss handler. The routing decision ‘which decode worker owns this session’ quietly becomes as consequential as any load-balancing choice in the stack, because moving a long conversation means moving gigabytes.