Why architecture matters here
The economics are the whole motivation. An LLM call has real marginal cost — tokens in and out, priced per million — and real latency, often a second or more for a substantial answer, longer if the agent takes tool-calling steps. In production, query distributions are heavily skewed: a support agent sees the same two hundred intents over and over, a documentation agent answers the same 'how do I' questions daily, an internal assistant fields the same policy lookups. When a large fraction of traffic is semantically repetitive, a cache that captures repetition by meaning can eliminate a big share of model calls, cutting both the bill and the p50 latency dramatically — a cached hit returns in the tens of milliseconds it takes to embed and search, versus the seconds a fresh generation costs.
But caching an agent's answers is riskier than caching a database row, and that risk is why the architecture matters. A row cache is either right or stale in an obvious way; a semantic cache can be subtly wrong. Two queries can be close in embedding space yet require different answers — 'how do I cancel my subscription' and 'how do I cancel my enterprise subscription' might sit a hair apart in vector space but have materially different processes. Serve the first answer to the second query and you have not saved money; you have given a confident wrong answer, which is worse than a slow right one. The entire design tension is between hit rate (serve more from cache, save more) and precision (never serve a wrong cached answer), governed primarily by the similarity threshold.
This makes semantic caching an explicitly tuned system rather than a drop-in. The threshold, the scoping, the freshness policy, and the verification step are all correctness controls, and every one is a dial between saving more and being wrong more. A team that treats a semantic cache as 'just add a cache' and skips the tuning ships an agent that is cheaper and occasionally, unpredictably, wrong — the failure mode that erodes user trust fastest.
There is a second, quieter reason the architecture matters: a semantic cache changes the shape of the latency distribution, not just its average. Fresh generations have a long, variable tail — a tool-calling agent might take one second or eight depending on the path it reasons through. Cache hits are near-constant and fast, so as the hit rate rises, the whole p90 and p99 of the system compress toward the embed-and-search floor. For an interactive agent, that tail compression is often worth as much as the token savings: users perceive a consistently snappy assistant rather than one that is sometimes instant and sometimes makes them wait. But this benefit is only real if the hits are correct, which loops back to the same constraint — precision is the price of every gain the cache offers, and it is bought with the threshold, the scoping, and the freshness policy, not with the cache alone.
The architecture: every piece explained
Top row: the lookup path. A user query in natural language is first passed through an embedding model that maps it to a vector capturing its meaning — the same model must be used for storing and for lookup, or the geometry is meaningless. That vector goes to a vector search over the cache index, which returns the nearest stored entry and its similarity score (usually cosine). The threshold gate is the decision point: if similarity is at or above τ, it is a hit; otherwise a miss. Everything downstream hinges on where τ sits.
Middle row: the two outcomes and the write path. On a cache hit the stored answer is returned directly — no model call, tens of milliseconds. On a cache miss the request proceeds to the LLM or the full agent, which produces a fresh answer; that answer, together with the query text, its embedding vector, and metadata, is written back as a new cache entry so the next similar query hits. Each entry carries TTL and invalidation information: a time-to-live after which it expires, and hooks to purge it when the underlying facts change (a policy update, a price change) so the cache never serves an answer that has since become false.
Bottom rows: the correctness controls. Scope keys partition the cache so a lookup only matches entries in the same context — the same tenant, the same user's permission set, the same tool or document version — because an answer correct for one scope can be wrong or a security leak in another. Verification is an optional but valuable guard: for borderline hits, a cheap check (a small model or a rule) confirms the cached answer actually fits the new query before serving it, catching the close-but-different case the raw threshold misses. The ops strip is the tuning surface: hit rate (are we saving enough?), false-hit sampling (are hits actually correct?), threshold tuning against that sampling, and staleness alerts when TTLs or invalidations are not keeping pace with the truth.
End-to-end flow
Trace a day in a support agent's semantic cache, scoped per tenant. The cache is warm from prior traffic. A user types 'i can't log in, forgot my password'. The system embeds it and searches within that tenant's scope; the nearest entry is a prior 'how do I reset my password' answer at cosine similarity 0.94, above the τ of 0.90. Hit. The stored step-by-step reset instructions return in 30 milliseconds, no model call, no token cost. The user is happy and the agent's bill is untouched for this turn.
A minute later a different user asks 'how do I reset my password for the admin console specifically'. Embed, search: nearest entry is again the generic reset answer, but similarity is 0.87 — below τ. Miss, correctly, because the admin console has a different flow, and the threshold protected us from serving the generic answer to a specific question. The request goes to the full agent, which produces the admin-specific instructions; that new (query, vector, answer) is stored, so the next admin-reset question will hit. This is the cache learning the shape of the tenant's real question distribution over time — hit rate climbs as coverage grows.
Now the correctness machinery earns its keep. At noon the tenant changes its password policy — resets now require MFA re-enrollment. An invalidation hook fires on the policy-update event and purges every cache entry tagged with the password-reset topic for that tenant. The stale generic answer is gone; the next reset question misses, regenerates against the new policy, and re-populates the cache with the correct, MFA-aware answer. Without that invalidation, the cache would have cheerfully served the old instructions for the rest of the TTL — cheap, fast, and wrong.
Finally the verification guard catches a subtle trap. A user asks 'how do I cancel my enterprise plan'; the nearest entry is the consumer 'how do I cancel my plan' answer at 0.91, just over τ. Rather than serve it blindly, the borderline hit triggers a cheap verification check that notices the 'enterprise' qualifier changes the process (enterprise cancellations route through an account manager). Verification vetoes the hit, the request regenerates the correct enterprise answer, and a new scoped entry is stored. On the dashboards the day reads well: a 60% hit rate saving roughly that share of model calls, a false-hit sample rate near zero thanks to the threshold and verification, and a clean staleness line where the policy invalidation swept the old answers exactly when the policy changed.