Why architecture matters here
Prompt caching is architectural because it changes the unit of work the model charges for from 'the whole prompt, every time' to 'only what is new since last time', and capturing that change requires structuring prompts around a stable prefix. The economics of LLM serving make prefill a first-order cost: processing input tokens consumes compute proportional to their number, and applications front-load enormous amounts of stable context — instructions, personas, examples, schemas, documents — that does not change between requests. Without caching, that stable context is a tax paid on every call. With caching, it is a one-time cost amortized across every request that reuses it. The architectural move is to recognize which parts of a prompt are stable and shared, and to arrange the prompt so that those parts form a cacheable prefix.
The defining constraint is that caching operates on a prefix and matches exactly, and this single fact dictates prompt design. The model's attention state at a given position depends on every token before it, so the cached state for a prefix is valid only if the request's tokens are identical up to that point. The cache is reusable from the start of the prompt up to the first token that differs from what was cached; everything from that divergence onward must be computed fresh. This is why the ordering of a prompt is suddenly load-bearing: the most stable content — the system prompt, fixed instructions, static examples — belongs at the very top, and anything that varies per request — the user's specific question, dynamic data, timestamps, ids — belongs at the bottom. Get this backwards and the cache is useless: a per-request token near the top means the divergence point is near the top, so almost nothing is cacheable.
The second architectural fact is that the cache is bounded in time, so hit rate depends on traffic patterns, not just prompt structure. Cached prefix state occupies scarce memory and is not kept indefinitely; it expires after a TTL and can be evicted under pressure. A prefix reused many times within the window is cached effectively; a prefix used once, or reused only after the window lapses, gets recomputed and provides no benefit. This couples the value of caching to your request rate and locality: high-traffic endpoints with a shared system prompt cache beautifully because the prefix is constantly refreshed within its TTL, while low-traffic or highly personalized workloads may see the cache expire between uses. Designing for caching therefore means thinking about which prefixes are hot enough to stay warm.
The subtlest architectural point is that caching must be transparent to correctness — the cached result must be identical to recomputing — and this is what makes it safe to adopt aggressively. Because the KV state is a deterministic function of the prefix tokens, loading it produces exactly the output the model would have produced by recomputing; caching is a pure performance optimization with no effect on results. That property is why prompt caching is not a lossy approximation like semantic caching, which returns a previous answer for a similar query and can be wrong. Prompt caching reuses the model's own internal work on an identical prefix and then continues generating normally from the new tail, so the model still fully processes the novel part of every request. The architectural implication is that you can and should cache maximally — there is no accuracy cost to a hit — and the only real engineering work is arranging prompts and traffic so that hits happen often. That combination of large upside and zero correctness risk is what makes prefix caching one of the highest-leverage optimizations in LLM application design.
The architecture: every piece explained
The top row is the lookup path. A prompt arrives, conceptually split into a stable prefix and a request-specific tail. The system derives a cache key from the prefix tokens — a hash of the exact token sequence — because the cache is addressed by content, not by name. That key looks up the prefix KV cache, the stored attention state (key/value tensors) the model produced when it last processed this prefix. If the state is present, prefill runs only over the new tail, appending its state to the loaded prefix state instead of rebuilding everything. The essential shape is: hash the prefix, load its state, compute only what is new.
The middle row is the two outcomes and their bookkeeping. On a cache hit, the stored prefix state is reused and the model skips reprocessing potentially thousands of prefix tokens — this is where the latency and cost savings come from. On a cache miss, the prefix has not been seen (or has expired), so the model computes its state from scratch and stores it for next time, paying full price on this request to benefit future ones. TTL and eviction govern the cache's lifetime: cached state expires after a configured window and can be evicted under memory pressure, so a prefix must be reused within the window to stay warm. After prefill, decode proceeds normally, generating output tokens one at a time from the combined state — decode is unaffected by caching; only the prefill of the prefix is saved.
The third row holds the two rules that make caching effective. Prefix ordering is the design discipline: put stable content first and variable content last, so the cacheable prefix is as long as possible and the per-request divergence happens as late as possible. The exact-match rule is the hard constraint behind that discipline: caching matches the token sequence exactly, so a single differing token — a changed word in the system prompt, an injected timestamp, a reordered example — moves the divergence point earlier and shrinks or eliminates the cached prefix. There is no fuzzy or partial matching; either the tokens are identical up to a point or they are not.
The ops strip names what to measure. Cache hit rate is the headline metric — it directly determines how much you actually save — and it is a function of prefix stability and traffic locality. Prefix stability tells you whether your prompts are constructed to be cacheable or are being accidentally busted. TTL versus traffic tells you whether hot prefixes stay warm between uses or expire in the gaps. And cost/latency savings is the outcome you are optimizing: the reduction in time-to-first-token and in billed input processing that the cache delivers, which is what justifies the effort of structuring prompts around it.
End-to-end flow
Walk a warm request. An application sends a prompt whose first several thousand tokens are a fixed system prompt plus a stable block of few-shot examples, followed by the user's specific question at the end. The system hashes the stable prefix, finds its KV state already cached from a prior request, and loads it. Prefill now runs only over the short user question at the tail, not the thousands of prefix tokens. The model produces its combined attention state quickly and begins decoding. Time-to-first-token drops sharply because the expensive prefix prefill was skipped, and the billed input cost for the cached portion is a fraction of the full price. The user's novel question was still fully processed — caching saved the redundant work, not the necessary work.
Now a cold request. The same application deploys a new version of its system prompt. The prefix tokens change, so the cache key changes, and the lookup misses. The model computes the full prefix state from scratch, pays the full prefill cost on this request, and stores the new state under the new key. The next request with the same new system prompt hits the freshly stored state and is fast again. This shows the amortization at work: a prefix change costs exactly one full recomputation, after which the new prefix is warm. It also shows why frequent prompt churn is expensive — every edit to the stable prefix forces a cold miss and restarts the amortization.
Now the self-inflicted failure: cache busting. A developer, wanting to log each call, prepends a per-request request-id or the current timestamp to the very top of the prompt, above the system prompt. Now every request's prefix diverges at the first token, so the divergence point is at position zero and nothing downstream is cacheable — the cache hit rate collapses to zero and every request pays full prefill, even though the enormous system prompt below is byte-for-byte identical across requests. Nothing errors; the model still works correctly; the only symptom is that costs and latency quietly return to their uncached levels. The fix is purely structural: move the variable request-id to the bottom of the prompt, below all the stable content, so the long prefix above it becomes cacheable again. This class of bug — an innocent per-request value placed too high — is the single most common reason a team's caching 'isn't working'.
Finally, the TTL-and-traffic interaction that governs real hit rates. Consider two endpoints sharing the same well-ordered system prompt. The high-traffic endpoint receives requests every few seconds, so the shared prefix is constantly re-referenced within its TTL and stays perpetually warm — near-100% hit rate after the first request. The low-traffic endpoint receives a request every few minutes, longer than the TTL, so the prefix expires between requests and each one is a cold miss despite being structurally identical. The architectural lessons are that caching rewards locality and that you can engineer locality: routing requests that share a prefix to the same server keeps that server's cache warm, batching or co-scheduling similar requests raises effective hit rate, and for critical low-traffic prefixes you can keep them warm deliberately. The performance of prompt caching is therefore not just a property of your prompts but of how your traffic is shaped and routed — which is why hit rate, not merely prompt structure, is the metric that tells you whether the architecture is paying off.