Why architecture matters here
Batching fails on starvation (long prompts block short decodes), fragmentation (KV cache holes), and preemption thrashing. Architecture matters because scheduler + KV cache + fairness together decide latency + throughput.
The architecture: every piece explained
The top strip is the runtime. Request queue incoming. Scheduler add/drop per step. Paged KV cache block-based. Iteration step one token per running request.
The middle row is control. Preemption swap to CPU or recompute. Fairness policy prompt vs decode weight. Prefill vs decode — different bounds. Chunked prefill interleaves.
The lower rows are ops. Metrics TTFT + TPS + goodput. Cancellation user disconnect. Ops — SLO tiering + eviction.
End-to-end flow
End-to-end: 40 requests running. Scheduler admits 3 more each iteration if KV cache has room. Long-prompt prefill runs chunked with decode; decodes not blocked. Cancelled request evicts its blocks; new request fills. Preemption on OOM swaps oldest. Metrics show TTFT p50 200ms, TPS stable.