Why architecture matters here
The KV cache matters because it converts the fundamental economics of transformer inference. Without it, generating a 500-token response to a 2,000-token prompt would recompute attention over the growing sequence at every step — quadratic work that would make interactive generation impossibly slow. With it, prefill does one big parallel pass over the prompt, and each subsequent decode step is a small, cheap operation. But the cache is not free: it stores, for every layer, for every attention head, for every token so far, a key vector and a value vector. For a modest SLM that is often tens to hundreds of kilobytes per token, which means a single long conversation can occupy hundreds of megabytes of scarce GPU high-bandwidth memory (HBM). The cache, not the model weights, is usually what runs you out of memory.
This reframes the whole serving problem. Decode is memory-bandwidth-bound: each step reads the entire KV cache and the model weights from HBM to compute one token, so throughput is gated by how fast you can move that memory, not by raw FLOPs. That is why batching helps enormously — processing many sequences together amortizes the weight reads across more useful work — and why the capacity to hold many sequences' KV caches simultaneously directly determines how many requests you can serve per GPU. Every byte you waste on cache fragmentation or over-allocation is a request you cannot batch.
The architectural consequences cascade. If you allocate a contiguous cache buffer sized for the maximum context length per request, you waste most of it (most requests are short) and fragment the rest, so you serve far fewer concurrent sequences than the hardware could hold. If you can instead manage KV memory in small fixed-size pages, share pages across requests with identical prefixes, quantize the stored keys and values, and evict or recompute under pressure, you can pack three to five times more sequences onto the same GPU. The KV cache is where SLM serving efficiency is won or lost, which is why modern inference engines are, at their core, KV-cache memory managers.
The architecture: every piece explained
Top row: the two phases. Prefill takes the full prompt and runs it through the model in one parallel forward pass, computing and storing the K and V for every prompt token at every layer — this is compute-heavy but happens once and sets up the cache. The KV cache itself is the per-layer, per-head store of those key and value tensors, indexed by token position. The decode step generates exactly one token: it computes the query, key, and value for the newest position, appends the new K and V to the cache, and runs attention of the single new query against all cached keys and values to produce the next token's logits. Decode repeats until an end-of-sequence token or a length limit, appending one KV entry per step.
Middle row: memory management, which is where the real architecture lives. Instead of one contiguous buffer per sequence, modern engines use paged blocks: KV memory is carved into small fixed-size pages (say 16 tokens' worth), and a sequence's cache is a list of pages that need not be contiguous — the same idea as virtual memory paging in an operating system. A per-sequence block table maps logical token positions to physical page addresses, so growing a sequence just allocates another page from a free list with near-zero fragmentation. Prefix sharing falls out naturally: if two requests share a common prompt prefix (a system prompt, a few-shot preamble), they can point their block tables at the same physical pages for that prefix, computing it once and storing it once. Eviction handles pressure: when memory is full, the engine can swap a paused sequence's pages to host memory, drop and later recompute them, or preempt the lowest-priority request.
Bottom rows: the density multipliers. Quantized KV stores keys and values in int8 or fp8 instead of fp16, roughly halving or quartering cache footprint at a small, usually acceptable quality cost — decisive for long-context SLMs. Continuous batching (a.k.a. in-flight batching) is the scheduler that packs many sequences into each forward pass and, crucially, admits new requests and retires finished ones between decode steps rather than waiting for a whole batch to complete — so a fast request does not wait behind a slow one, and the GPU stays saturated. The ops strip names the dials that matter: the HBM budget split between weights and cache, the page/block size, the cache hit rate for prefix sharing, and the preemption policy under overload.
End-to-end flow
Follow two requests through a paged, continuously batched SLM server. Request A arrives with a 1,800-token document-QA prompt that begins with a 300-token shared system preamble; request B arrives moments later with the same preamble and a short 120-token question. The scheduler admits A first. Prefill runs A's 1,800 tokens in one pass, allocating pages from the free list and writing the block table; the 300-token preamble occupies its own pages, which the engine hashes and registers in a prefix-sharing table.
When B is admitted, the prefill planner checks the prefix table, finds A's preamble pages match B's, and points B's block table at those existing physical pages — B's prefill only has to compute the 120 novel question tokens, and the preamble's KV is neither recomputed nor re-stored. That is a real saving in both latency and memory: on a busy server where hundreds of requests share a system prompt, prefix sharing can cut aggregate prefill cost by a large fraction and free enough KV memory to raise the concurrent-sequence count sharply.
Now decode. Both sequences are in the active batch. Each decode step runs one forward pass over the whole batch: for every sequence it computes one new query/key/value, appends the K/V to that sequence's next page (allocating a fresh page when the current one fills), and attends against the sequence's cached K/V via the block table's page list. Continuous batching means that when B finishes after 60 tokens, its pages are freed immediately and a newly arrived request C is admitted into the very next step — the GPU never idles waiting for A to finish its longer generation. Quantized KV keeps each stored K/V in fp8, so the server holds perhaps twice as many sequences as an fp16 cache would.
Then the stress case: a burst of long-context requests arrives and the free page list runs dry mid-decode. The engine's preemption policy kicks in. It selects a lower-priority sequence, swaps its KV pages out to pinned host memory (or, cheaper for short sequences, drops them and marks the sequence for recomputation), frees those physical pages for the hot requests, and continues. When pressure eases, the swapped sequence's pages are restored and it resumes exactly where it paused. The user of the preempted request sees a brief stall, not an error; the throughput of the whole fleet stays high because the engine never over-committed memory it did not have. This graceful degradation under memory pressure — enabled entirely by paging the KV cache — is the difference between a server that thrashes and one that bends.