Why architecture matters here
Prompt caching matters because it's one of the highest-leverage, lowest-risk optimizations for LLM applications -- significant cost and latency reduction for the common case of repeated context, with no quality tradeoff. LLM applications very commonly send repeated context: a lengthy system prompt on every request (the same instructions, persona, tool descriptions -- sent every call), retrieved documents in RAG (the same documents across follow-up questions), few-shot examples (the same examples in every prompt), or conversation history (the same history prefix as a conversation continues). Processing this repeated context (the prefill -- computing the KV cache for the prompt) is expensive (the prefill compute, proportional to the context length) and repeated (recomputed every request, even though it's the same). Prompt caching eliminates this waste (caching the prefix's computation, reusing it) -- providing significant cost reduction (the cached prefix billed at a fraction) and latency reduction (the prefix prefill skipped -- faster response). And it's risk-free (the cache is exact -- the same prefix, so the same computation -- no quality tradeoff, it's the identical result just cached). For LLM applications with repeated context (most applications), prompt caching is a major optimization (cost, latency) with no downside -- making it one of the first optimizations to apply, and understanding it (how it works, how to structure prompts for it) is essential to efficient LLM applications.
The cache-alignment requirement is the crucial architectural constraint, and it's what determines whether caching works. Prompt caching caches a prompt prefix (the computed representation for the beginning of the prompt -- the KV cache is inherently prefix-based, since the model processes the prompt left-to-right and the KV cache for a position depends on all preceding positions). And the cache matches on the exact prefix (byte-for-byte -- the cached representation is for a specific prefix, reused only when the request's prefix is identical). So caching requires cache alignment: the prompt must be structured with the stable, repeated content first (the cacheable prefix -- the system prompt, fixed context) and the variable content last (the non-cached suffix -- the specific query). If the prefix is identical across requests (the stable content first, byte-for-byte the same), the cache hits (the prefix's computation reused). If anything in the prefix varies (a timestamp in the system prompt, a request ID, reordered content -- making the prefix different), the cache misses (the prefix doesn't match -- everything recomputed, no benefit). This is the crucial constraint: caching works only for identical prefixes, so prompts must be cache-aligned (stable prefix, variable suffix), and any volatility in the prefix breaks caching. The classic mistake (putting something volatile -- a timestamp, dynamic content -- in the prefix, breaking caching silently -- the requests all missing the cache, no benefit, unaware) illustrates the importance. Understanding the cache-alignment requirement (identical stable prefix for cache hits, volatility breaking it) -- and structuring prompts accordingly -- is understanding how to make prompt caching actually work.
And the hit-rate-plus-freshness reality is the operational core, balancing reuse against freshness. Hit rate is the value metric: how often requests hit the cache (reuse the cached prefix) vs miss (recompute) -- a high hit rate means most requests benefit (reusing the cache -- cost and latency savings), a low hit rate means little benefit (mostly recomputing). The hit rate depends on cache alignment (identical prefixes -- so structuring prompts for alignment maximizes hits) and the cache's TTL/eviction (the cache is retained for a TTL -- if a repeated prefix is requested within the TTL, it hits; if the TTL expires, the next request misses and re-caches). Freshness vs reuse: the cache TTL balances freshness (a shorter TTL means the cache is fresher -- re-computed more often, but also missed more) against reuse (a longer TTL means more reuse -- more hits, but the cache held longer). For prompt caching, the cached content (the prefix computation) is deterministic (the same prefix always computes the same -- so there's no staleness concern from the content changing; the cache is exact), so the TTL is mainly about cache retention (how long to keep the cache for reuse -- longer for more reuse, subject to the cache's capacity/eviction). Maximizing the hit rate (via cache alignment -- identical prefixes -- and appropriate TTL/retention) is the operational goal (more hits = more cost/latency savings). Understanding the hit rate (the value metric, driven by cache alignment and retention) is understanding how to measure and maximize prompt caching's benefit.
The architecture: every piece explained
Top row: the mechanism and benefits. KV cache reuse: the model's computed representation (the KV cache) for a prompt prefix is cached and reused -- so the prefix's expensive computation (the prefill) is done once and reused (not recomputed per request). Prefix matching: the cache matches on the shared prompt prefix (the byte-for-byte identical beginning of the prompt) -- reused when the request's prefix matches the cached prefix. Cost reduction: cached prefix tokens are billed at a fraction of the normal rate (since they're not recomputed -- just the cached representation reused) -- significant cost savings for the repeated prefix. Latency reduction: the prefix prefill is skipped (the cached representation used instead of recomputing) -- so the response starts faster (the expensive prefix processing cached, not on the critical path).
Middle row: alignment and control. Cache-aligned prompts: structuring prompts with the stable content first (the cacheable prefix -- system prompt, fixed context) and the variable content last (the non-cached suffix -- the specific query) -- so the prefix is identical across requests (cacheable). Cache TTL and eviction: the cache is retained for a TTL (and subject to eviction -- capacity limits) -- balancing reuse (longer TTL = more hits) against retention cost -- the cache lifecycle. Explicit vs automatic: caching can be automatic (the provider caches prefixes automatically -- transparent) or explicit (the application marks cache points -- controlling what's cached) -- depending on the provider. Hit rate: the value metric -- how often requests hit the cache (reuse) vs miss (recompute) -- high hit rate = high benefit; driven by cache alignment and retention.
Bottom rows: pitfalls and applications. What breaks caching: volatile prefix content -- anything in the prefix that varies (a timestamp, request ID, reordered content, dynamic data) makes the prefix different (a cache miss -- everything recomputed) -- the classic mistake (volatile content in the prefix silently breaking caching -- all misses, no benefit, unaware). Use cases: system prompts (the same lengthy system prompt on every request -- cached), RAG (the same retrieved documents across follow-ups -- cached), few-shot examples (the same examples in every prompt -- cached), conversation history (the history prefix -- cached as the conversation continues) -- the common repeated-context cases. The ops strip: cache alignment (structuring prompts for caching -- stable prefix, variable suffix, no volatile prefix content -- the primary requirement for caching to work), hit rate (measuring the cache hit rate -- the value metric; a low hit rate indicates poor alignment or insufficient repetition), and cost attribution (attributing the cost savings from caching -- measuring the benefit, justifying the alignment effort).
End-to-end flow
Trace prompt caching reducing cost and latency. An application has a lengthy system prompt (detailed instructions, persona, tool descriptions -- say 2000 tokens) sent on every request, plus the specific user query (varying). Cache-aligned: the prompt is structured with the system prompt first (the stable, cacheable prefix -- identical every request) and the user query last (the variable, non-cached suffix). On the first request, the system prompt prefix is processed (prefilled -- computing its KV cache) and cached. On subsequent requests (with the same system prompt prefix), the cache hits: the system prompt's computation is reused (not recomputed -- the cached KV cache used), so only the user query (the varying suffix) is processed fresh. The benefits: cost -- the 2000-token system prompt prefix is billed at a fraction (cached, not recomputed) on every cached request (major savings, since the system prompt dominates the prompt size); latency -- the system prompt prefill is skipped (the cached representation used), so the response starts faster (the expensive 2000-token prefill cached, not on the critical path). The cache alignment (stable system prompt prefix, variable query suffix) enabled the caching -- reducing cost (cached prefix billed less) and latency (prefix prefill skipped) for the repeated system prompt.
The cache-breaking and hit-rate vignettes show the pitfalls and measurement. A cache-breaking case: a developer adds a timestamp to the system prompt (e.g., 'Current time: ...' -- for context) -- putting volatile content (the timestamp, different every request) in the prefix. This breaks caching: the prefix now varies (the timestamp different each request), so every request misses the cache (the prefix doesn't match -- the system prompt recomputed every time, no caching benefit). And it's silent (the requests just miss the cache -- no error, no benefit, and the developer unaware the caching broke). The fix: remove the volatile timestamp from the prefix (or move it to the variable suffix -- after the cacheable prefix) -- restoring the identical prefix (cacheable) -- the caching working again. The lesson: volatile content in the prefix silently breaks caching (a common, costly mistake). A hit-rate case: the team monitors the cache hit rate -- confirming most requests hit (high hit rate -- the caching working, the repeated system prompt cached) -- and would catch a drop (indicating broken caching -- e.g., someone adding volatile prefix content) via the hit-rate metric.
The RAG and cost-attribution vignettes complete it. A RAG case: in a conversation over retrieved documents, the same documents (the retrieved context) are sent across follow-up questions. Cache-aligned (the documents as the stable prefix, the follow-up question as the variable suffix), the documents' computation is cached and reused across the follow-ups -- so the (potentially large) document context isn't recomputed per follow-up (cost and latency savings across the conversation) -- the caching benefiting the RAG conversation (the repeated document context cached). A cost-attribution case: the team attributes the cost savings from caching (measuring the cached vs non-cached token costs -- the cached prefix tokens billed at a fraction, quantifying the savings) -- confirming the caching's benefit (significant cost reduction from the cached system prompts and RAG documents) and justifying the cache-alignment effort. The consolidated discipline the team documents: use prompt caching for repeated context (system prompts, RAG documents, few-shot examples, conversation history -- the common repeated-context cases -- a high-leverage, risk-free optimization), cache-align prompts (stable content first as the cacheable prefix, variable content last -- the primary requirement), avoid volatile prefix content (no timestamps, IDs, reordered content in the prefix -- it silently breaks caching), monitor the hit rate (the value metric -- catching broken caching), attribute the cost savings (quantifying the benefit), and understand what breaks caching (prefix volatility) -- because prompt caching reuses the computed representation of repeated prompt prefixes, providing significant cost (cached prefix billed less) and latency (prefix prefill skipped) reduction for the common case of repeated context, risk-free, requiring only cache alignment (stable prefix, no volatile prefix content) to work.