Why architecture matters here
Disaggregation architecture matters because prefill and decode have fundamentally different bottlenecks. Prefill runs a single large forward pass and is compute-bound; decode reads all KV cache and is memory-bandwidth-bound. Running both on the same GPU pool means one always leaves capacity on the table for the other.
Cost impact is substantial. A disaggregated stack can serve 2-3x more concurrent requests than a colocated one at the same p99 latency. On expensive H100 GPUs, this savings is meaningful.
Reliability comes from separate SLO targets. TTFT (time to first token) is a prefill concern; ITL (inter-token latency) is decode. Autoscaling each pool independently against its SLO is cleaner than juggling both on one pool.
The architecture: every stage explained
Walk the diagram top to bottom.
Request. User's prompt plus generation params (max tokens, temperature).
Router. Routes to prefill pool. May pick specific prefill instance based on load or model shard placement.
Prefill Pool. GPUs optimized for compute-bound work: many SMs, tensor cores. H100 or A100. Runs the prompt through the model in one forward pass; produces KV cache for the entire prompt plus the first output token.
Prefill computation. Attention is O(n²) in prompt length; runs efficient with Flash Attention. Output: KV cache + first token.
KV Cache Transfer. The KV cache (potentially GBs for long prompts) transfers from prefill GPU to decode GPU over NVLink (same node) or InfiniBand/RoCE (across nodes). This is the critical path.
Decode Pool. GPUs optimized for memory bandwidth: same H100 or A100, but different loading — each GPU serves many concurrent decodes because decode is memory-bound not compute-bound. Continuous batching packs many requests together.
Continuous Batching. vLLM/TRT-LLM iteration-level scheduling. Each token step processes a mix of active decodes plus new arrivals.
Response Streaming. SSE or gRPC streams tokens back to the client as they generate.
Separate SLO targets. TTFT SLO on prefill pool (say, p99 < 500ms). ITL SLO on decode pool (say, p99 < 50ms per token).
Autoscale independently. Prefill demand is bursty (long prompts arrive); decode demand is steadier. Different HPA policies per pool.
End-to-end disaggregated request flow
Trace a request. User sends prompt of 4k tokens. Router picks a prefill instance with capacity. Instance runs the 4k prompt through the model in one forward pass.
Prefill instance produces KV cache: for 70B model at FP16 with 4k prompt, KV cache is roughly 1.5 GB per request.
Prefill instance also produces the first output token.
KV cache transfer initiates: 1.5 GB from prefill GPU to decode GPU. Over NVLink (900 GB/s), takes ~2ms. Over InfiniBand (200 Gbps effective), takes ~60ms.
Decode instance receives the KV cache and the first token. Adds this request to its continuous batch. Next token generation includes this request among the others.
Decode continues token by token. Each step reads all KV caches for all active requests, produces next tokens, appends new KV entries. Streaming responses go back to clients.
User sees first token quickly (prefill was compute-bound but fast on H100); subsequent tokens stream at 30-60 tokens/sec (decode packs many requests).
Meanwhile, another user's shorter prompt hits a different prefill instance in parallel. Prefill pool sizes for parallelism.