Why architecture matters here
The architecture matters because inference is the dominant cost of an LLM application, in both money and latency, and a large share of production traffic is repetitive. Support bots, documentation assistants, and internal knowledge tools field the same handful of intents over and over, phrased differently each time. Every one of those repeats, absent a semantic cache, is a full model call: tokens billed, GPU time consumed, seconds of latency imposed on the user. A cache that recognizes semantic repeats converts that recurring cost into a one-time cost plus cheap lookups.
It matters for latency as much as cost. A vector search over even a few hundred thousand cached queries returns in single-digit milliseconds; a model generation takes hundreds to thousands. For the queries a semantic cache can serve, it is not a marginal speedup but two or three orders of magnitude — the difference between an answer that feels instant and one the user waits on. In a conversational product, that latency gap is felt directly.
But the architecture matters most because it introduces a failure mode that ordinary caching does not have: the risk of serving a semantically similar but wrong answer. An exact cache is always correct — if the key matches, the value is by definition right. A semantic cache trades that certainty for coverage, and the design has to actively manage the resulting false-hit risk with thresholds, metadata scoping, and monitoring. Treating a semantic cache like a normal cache — 'a hit is a hit' — is how you ship confidently wrong answers to users.
Finally, it matters because the cache sits at a trust boundary. Cached answers are reused across users, so a naive shared cache can leak one user's personalized or tenant-specific answer to another, or serve a stale answer after a policy changes. The architecture has to encode who an answer is valid for and how long it stays valid. A semantic cache is not just a performance component; it is a correctness- and privacy-sensitive one, and its design has to reflect that.
The architecture: every piece explained
The read path has four stages. First, an embedding model encodes the incoming query into a fixed-length vector. This should be the same model used to embed cached queries, and ideally one tuned for short-text semantic similarity; the quality of every hit decision inherits from this encoder. Second, a vector index (FAISS, HNSW, a vector database) returns the top-k nearest stored query vectors by cosine similarity. Third, a threshold gate compares the top match's similarity against a cutoff τ: at or above τ it is a hit; below, a miss. Fourth, on a hit, the stored answer associated with that query is returned.
The write path runs on every miss. After the model generates a fresh answer, the system stores the query embedding, the answer, and a bundle of metadata: a TTL, the tenant or user scope, the prompt/version the answer was generated under, timestamps, and often the token cost saved. This metadata is what turns a raw nearest-neighbor lookup into a correct cache. The scope fields ensure a lookup only considers entries valid for this requester; the version field lets you invalidate every entry generated under an old system prompt in one stroke.
The threshold is the single most consequential parameter. It is not a universal constant — it depends on the embedding model's geometry and on how costly a false hit is for your domain. A factual support bot where a wrong answer is expensive wants a conservative, high threshold that only fires on near-paraphrases; a low-stakes suggestion feature can afford a looser one for more coverage. Many systems calibrate τ empirically against a labeled set of query pairs, choosing the point that maximizes hit rate while holding false hits below a tolerance.
Scoping and isolation are structural, not optional. A per-tenant (or per-user, for personalized answers) partition of the cache prevents one requester's answer from ever being served to another. This can be a metadata filter applied during search, or physically separate indexes per tenant. For answers that depend on the asker's permissions or data, the scope must be narrow enough that a cache hit is only ever valid for someone who would have received the same answer anyway. Getting scoping wrong turns a performance optimization into a data-leak vector.
It is worth being explicit about what an entry stores, because it is more than a string. A well-formed entry pairs the query embedding (the search key) with the answer text, but also with the generation context that produced it: which prompt version, which retrieved documents or their identifiers, which model, and which tenant/user scope. That context is what makes correctness auditable and invalidation surgical. When a source document changes you can expire exactly the entries that cited it; when you bump the system prompt you can expire exactly the entries generated under the old one; when you audit a suspected false hit you can see the full provenance of the answer that was served. An entry that stores only 'question vector → answer' works until the day something upstream changes, at which point you have no way to know which cached answers are now wrong. Rich metadata is the difference between a cache you can operate surgically and one you can only flush wholesale.
End-to-end flow
Follow a query through the cache. A user asks 'can I get a refund on a digital purchase?' The service embeds the query with the shared encoder, producing a vector, and searches the tenant-scoped vector index for the nearest prior queries. The top match is 'how do refunds work for digital items?' at cosine similarity 0.94, comfortably above the configured threshold of 0.90.
Because it is a hit within the same tenant scope and the entry is not expired, the service returns the stored answer directly. No embedding of context, no prompt assembly, no model call — a few milliseconds of vector search and a lookup. The event is logged: a hit, the matched entry, the similarity score, and the estimated tokens and dollars saved. That log is the raw material for later hit-rate and false-hit analysis.
Now a different user asks 'what's the warranty on physical goods?' The embedding lands far from any refund entry; the nearest neighbor's similarity is 0.71, below the threshold. This is a miss. The service falls through to the normal pipeline: assemble the prompt (perhaps with retrieval), call the model, stream the answer to the user. On completion, it writes a new cache entry — the warranty query's embedding, the generated answer, the tenant scope, a TTL, and the current prompt version — so the next warranty question hits.
Later, the team updates the refund policy and bumps the prompt version. On the next deploy, the invalidation job drops (or marks stale) every cache entry generated under the old version, so refund questions temporarily miss and regenerate against the new policy, repopulating the cache with correct answers. Throughout, a background monitor samples a fraction of hits, re-runs the model on the original query, and compares — measuring the false-hit rate that tells operators whether the current threshold is safe. The cache is thus a living loop: serve on hit, generate and store on miss, invalidate on change, and continuously audit that hits are actually correct.