Why architecture matters here
Architecture matters here because KV memory management is the single largest lever on serving cost for a small model. SLMs are chosen precisely to be cheap and fast — running on a single modest GPU, at the edge, or many-per-host. If half your GPU memory evaporates into fragmentation, you have effectively bought a bigger, more expensive accelerator to run a small model badly. The difference between 45% and 96% memory utilization is the difference between batching eight sequences and batching thirty, which is a 3-4x swing in tokens per second per dollar on identical hardware.
The problem is genuinely hard because request lengths are unknown and unequal. You do not know at admission time whether a prompt will produce five tokens or five hundred, so any scheme that pre-commits memory guesses wrong constantly. Sequences also arrive and depart continuously, so the allocator faces the same fragmentation dynamics as a long-running OS heap: without a paging indirection, you get holes you cannot fill, and you are forced to either reject admittable requests or run compaction that stalls the GPU. Paging makes every block interchangeable, so a freed block from a finished chat is immediately usable by any other sequence, regardless of position or length.
Consider the arithmetic of the naive scheme to feel the pressure. If the model's maximum context is 4,096 tokens and you reserve that per sequence, then a request that generates 50 tokens uses barely more than one percent of its reservation, and the other ninety-nine percent is dead weight for the request's entire lifetime. With even a dozen such requests live at once, the majority of the KV pool is committed to memory that will never be written. Paging removes the reservation entirely: that 50-token request holds four blocks, full stop, and the rest of the pool stays available. The effective batch size the accelerator can sustain rises accordingly, and throughput rises with it, because in autoregressive decoding throughput is almost entirely a function of how many sequences you can process in parallel per step.
There is a second, subtler payoff. Many requests share content — the same long system prompt, the same few-shot examples, the same retrieved document prefix. In a contiguous world every sequence carries its own private copy of that prefix's KV. Under paging, those blocks can be shared by reference and only copied when a sequence diverges. For a fleet where every request begins with a 600-token policy preamble, prefix sharing alone can reclaim a large fraction of memory and skip recomputing the preamble's attention entirely. The architecture is what unlocks both fragmentation elimination and sharing at once.
It is worth being concrete about why this is not merely an optimization but a structural change in what the server can do. Because memory is allocated per block rather than per sequence, the scheduler is free to admit a request the instant a handful of blocks are available, rather than waiting for a slab large enough for the worst case. Short and long sequences coexist in the same pool without one starving the other, and the batch can grow and shrink continuously as sequences finish. That flexibility is what lets a single small model saturate its accelerator across a messy, real-world mix of prompt and generation lengths — the exact workload where contiguous reservation schemes waste the most.
The architecture: every piece explained
Top row: the control plane. The scheduler decides, each decoding step, which sequences run. It admits new requests when free blocks exist and preempts running ones when memory is exhausted. The block manager owns a free-list of fixed-size physical blocks in the GPU pool; allocation is popping a block off the list, freeing is pushing it back — both O(1), no compaction. Each sequence has a block table, a small array mapping logical block index (position / block_size) to a physical block id, plus a count of how many slots in the last block are filled. The GPU block pool is one big pre-allocated tensor sliced into equal blocks; nothing else on the device competes for it.
Middle row: the data plane. The attention kernel cannot assume contiguity, so it is written to gather: for a query token, it walks the sequence's block table, loads each physical block of keys and values in turn, and accumulates the attention output. This gather adds a layer of indirection versus a dense contiguous read, but block sizes are tuned so the kernel still moves memory in coalesced, cache-friendly chunks. Copy-on-write handles sharing: when two sequences reference the same prefix blocks and one of them needs to write into a shared block (because generation has reached it), the manager clones just that block and repoints the writer's table entry, leaving the readers untouched. Swapping and recompute are the pressure valves: under memory exhaustion a preempted sequence's blocks are either copied to CPU RAM (swap) or dropped and later recomputed from the prompt.
Bottom row: the throughput machinery. Continuous batching means the batch is not fixed for a request's whole lifetime; at every step finished sequences leave and waiting ones join, so the GPU never idles waiting for the slowest member of a static batch to finish. The prefix cache is prefix sharing made persistent and global: the KV blocks for a common system prompt are computed once, hashed, and kept resident so any new request beginning with that prefix reuses them instead of recomputing, turning a long shared preamble from a per-request cost into a one-time cost.
The ops strip is not decoration. GPU memory utilization tells you how much of the pool holds live tokens. Preemption rate tells you whether the scheduler is thrashing sequences in and out under load. Block-table depth and cache hit ratio tell you whether long contexts and prefix reuse are behaving. These four numbers, watched together, describe the health of the entire serving loop.
End-to-end flow
Trace two requests through the server. Request A arrives with a 600-token shared system prompt plus a 40-token user question. The scheduler admits it; the prefill pass computes KV for all 640 tokens. The block manager allocates 640 / 16 = 40 blocks and fills A's block table. Crucially, the first 600 tokens are the known system prompt: the prefix cache already holds those 38 blocks from an earlier request, so the manager points A's table at the cached physical blocks and only computes KV for the new user tokens — prefill work and memory both drop sharply.
Request B arrives moments later with the same system prompt and a different question. The scheduler admits it and points B's first 38 block-table entries at the identical cached blocks. A and B now physically share the preamble's KV. Decoding proceeds in a continuous batch: each step generates one token for A and one for B. When a sequence crosses a block boundary — its current block fills — the manager pops a fresh block off the free-list and appends it to that sequence's table. No pre-reservation; growth is exactly one block at a time.
Now load spikes: a burst of new requests exhausts the free-list. The scheduler must make room, so it preempts the lowest-priority running sequence. Its blocks are either swapped to CPU RAM or freed with a note to recompute later. The freed blocks go back on the list and the new arrivals run. When pressure eases, the preempted sequence is swapped back in or recomputed from its prompt and rejoins the batch. From the client's view this is invisible latency; from the server's view it is graceful degradation instead of an out-of-memory crash.
Request A finishes first — it hit an end-of-sequence token. The manager walks A's block table and returns every block that A uniquely owns to the free-list; the shared prefix blocks have a reference count, so they are not freed while B still points at them. B keeps generating, eventually reaches a point where it must write into a shared block, triggers copy-on-write for that one block, and continues on its private copy. When B finishes, its blocks return too, and the prefix cache retains the preamble blocks for the next request that starts the same way. Throughout, memory was allocated in tiny increments, shared aggressively, and reclaimed instantly — the paging discipline doing exactly what it does for an operating system.
Step back and count what the paging indirection bought across that trace. Two requests shared a 600-token preamble instead of holding two private copies, so preamble memory was paid once, not twice, and prefill for the second request skipped the shared blocks entirely. The load spike was absorbed by preemption instead of an out-of-memory crash, degrading gracefully. Growth happened one 16-token block at a time, so neither request over-reserved. And reclamation was reference-counted and immediate, so the moment a block stopped being needed it was available to any other sequence. Every one of these behaviors is impossible under contiguous per-sequence slabs; each falls out naturally once logical positions are decoupled from physical blocks through the block table.