Why architecture matters here
The architecture matters because prefill is where latency and cost concentrate for prompt-heavy workloads, and it is the part of the request that is most often redundant. Time-to-first-token — the pause a user feels before words appear — is dominated by prefill, and prefill scales with prompt length. A chat assistant with a 2,000-token system prompt and a 20-token user question spends the overwhelming majority of its prefill compute on tokens the user never wrote and that never change. Caching that prefix collapses time-to-first-token from 'read 2,020 tokens' to 'read 20 tokens', a difference a user perceives directly as the assistant feeling snappy rather than sluggish. On a small model where each request is cheap, the prefill of a shared prompt can still be the single largest slice of per-request GPU time, so eliminating it multiplies effective throughput.
The second forcing function is economics at scale. GPU time is the dominant cost of a serving fleet, and prefill of repeated prefixes is cost spent to compute the same answer over and over. A service handling millions of chats a day against one system prompt is, without prefix caching, paying to re-derive that prompt's attention state millions of times. With caching, it pays once per prefix per cache lifetime and reclaims the rest as free throughput headroom — the same hardware serves more users, or the same users at lower cost. Because the savings scale with how much prompt is shared and how often, prompt-heavy products (RAG with a fixed instruction block, agents with long tool descriptions, few-shot classifiers) see the largest wins.
The third reason is that prefix caching is not free correctness. The KV cache reused must correspond to exactly the same tokens through exactly the same model weights and tokenizer; a prefix that merely looks similar, or a model version bump that leaves stale blocks around, silently corrupts generation with attention state that belongs to a different context. The architecture therefore has to make the cache key precise — token-exact, model-version-scoped — and manage GPU memory carefully, because KV blocks compete for the same scarce HBM as the model weights and the active decoding batch. Getting the keying and the memory budget right is what separates a cache that speeds everything up from one that occasionally returns garbage or thrashes.
There is a subtler architectural payoff: prefix caching changes how you can structure prompts to maximize reuse. Once the cache exists, it pays to put everything stable at the front — instructions, tool schemas, retrieved documents shared across a session — and everything variable at the end, so the shared prefix is as long as possible and the divergent suffix as short as possible. This 'stable-prefix, variable-suffix' discipline is an emergent design rule that only makes sense because the cache rewards it, and teams that adopt it see hit rates and time-to-first-token improve without touching the model at all.
The architecture: every piece explained
Top row: turning a request into a cache lookup. The incoming request is a full token sequence — system prompt, conversation history, the new user turn. The prefix hasher walks the tokens in fixed-size blocks (say 16 tokens per block) and computes a rolling hash of each block chained to the previous, so the hash at block k uniquely identifies the exact token sequence from position zero through block k. This block-chained hashing is what lets two requests share a prefix of any length: they match block-for-block until the first block that differs. The radix / hash index maps each prefix hash to the cached KV blocks that hold that prefix's attention state; a radix tree is common because it naturally shares structure among prefixes with common roots. The block allocator manages KV memory in fixed pages, reference-counted so a block shared by ten requests is freed only when the last one releases it.
Middle row: hit, miss, and decode. On a cache hit, the longest matching prefix's KV blocks are found in the index, their reference counts incremented, and prefill for that span is skipped entirely — the model starts computing at the first unmatched token. On a cache miss (or a partial match), the model prefills the unmatched suffix, and the newly computed blocks are inserted into the index so the next request with the same prefix hits. The decode loop then generates output tokens, extending the sequence from the shared state; each generated token appends to KV blocks owned by this request alone, since the divergent suffix is not shared. Eviction (LRU) reclaims blocks when memory is tight, preferring cold prefixes that no active request references.
Bottom rows: the budget and the guards. The GPU HBM budget is the hard constraint — KV blocks share high-bandwidth memory with the model weights and the working set of the active batch, so the cache can only grow into whatever HBM the weights and in-flight requests leave free. Push it too far and you evict blocks a running request still needs, or you shrink the batch and lose decode throughput. The correctness guards keep reuse honest: the cache key must include the model version and tokenizer so a deploy invalidates stale blocks, and the prefix match must be token-exact rather than approximate. The ops strip names the signals that matter: cache hit rate, prefill compute saved, HBM pressure, and eviction churn.
End-to-end flow
Trace two chat requests through a serving node running the same small model behind a fixed system prompt. The first request arrives: system prompt (2,000 tokens) plus the user's question (18 tokens). The prefix hasher blocks and hashes the sequence; the radix index has nothing yet, so it is a full miss. The model prefills all 2,018 tokens, and as it does, each block of computed KV is inserted into the index under its prefix hash. Decode runs, the answer streams out, and when the request finishes, the suffix blocks (the question and the generated answer) are released, but the 2,000-token system-prompt prefix stays resident because a background policy pins hot shared prefixes.
Now the second request arrives moments later: the same 2,000-token system prompt plus a different 25-token question. The hasher walks the blocks and matches the index exactly through the end of the system prompt — 125 blocks of 16 tokens hit — then diverges at the first token of the new question. The node increments the reference counts on those 125 shared blocks and skips their prefill entirely; it prefills only the 25-token question. Time-to-first-token for this request is a fraction of the first request's, because the expensive part — reading and encoding 2,000 tokens of instructions — was already done and is simply reused. The GPU that would have spent its cycles re-encoding the system prompt instead spends them decoding output or serving another request in the batch.
Memory pressure builds as the day goes on. A conversation thread accumulates: each turn shares the growing history prefix with the next, so a ten-turn chat reuses the KV of the first nine turns and prefills only the tenth. This is prefix caching's second big win — not just shared system prompts across users, but shared history within a session. But every resident prefix consumes HBM, and HBM is finite. When the block allocator can no longer satisfy a new request, the LRU eviction policy reclaims the coldest unreferenced blocks — an idle user's stale conversation, a one-off request's leftover suffix — freeing pages without touching any prefix an active request still holds a reference to.
Then a deploy lands: a new model version rolls out. Because the cache key includes the model version, every block from the old version is now unreachable — no new request's key will match them — and they age out under LRU while the new version repopulates the cache from cold. For a few minutes hit rate dips and time-to-first-token rises as the new version re-prefills common prefixes, then the cache warms and latency settles back down. The whole cycle — miss, insert, hit, share, evict, invalidate — runs continuously, and the node's effective throughput tracks directly with how much of its prefill it manages to skip.