Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk

How a semantic cache decides a hit

Every semantic cache, whether it is GPTCache, a RedisVL SemanticCache, a LangChain cache backed by a vector store, or a few hundred lines in your own gateway, runs the same loop. The incoming request is normalised (often only the latest user message, sometimes the whole prompt), embedded into a vector, and used to query a vector index of previously answered requests. If the nearest stored entry is similar enough, its stored response is returned; otherwise the request goes to the LLM and the new pair is inserted.

def answer(request):
    key = normalise(request)                 # e.g. last user turn, lower-cased
    v = embed(key)                           # unit-length vector
    hit = index.nearest(v, filter=scope(request), k=1)
    if hit and hit.similarity >= THRESHOLD and not expired(hit):
        log_hit(request, hit)                # keep the score for audits
        return hit.response
    response = llm(request)
    index.insert(v, response, scope=scope(request), created=now())
    return response

The cache has no notion of meaning. It has a geometry and a threshold. Everything that goes wrong in production is a case where the geometry and the business meaning disagree, or where the stored response stopped being true. Note that this is a different thing from the prefix or prompt caching offered by inference servers and model APIs: those reuse computation for an identical token prefix and still generate a fresh answer, so they cannot return a wrong answer. A semantic cache reuses the answer, which is why it saves far more and can hurt far more.

The similarity arithmetic behind the threshold

Model a family of paraphrases as a centroid vector c plus noise: each phrasing is c + n, with noise of relative size σ, normalised to unit length. For two independent paraphrases the expected cosine is roughly 1 / (1 + σ²). With σ = 0.31, typical of a decent general-purpose embedding on short questions in this toy model, two phrasings of the same question land around 0.91. Now take a second question whose centroid sits at cosine ρ to the first: "can I return an ebook?" versus "can I return a paperback?". Its phrasings land around ρ / (1 + σ²) from the first family. With ρ = 0.96 that is about 0.88.

So the threshold is being asked to separate a distribution centred near 0.91 from one centred near 0.88, and both have spread because some paraphrases are terse and some are rambling. There is no threshold that admits most correct reuses and rejects all lookalikes; there is only a choice of where on the overlap you sit. Two levers change the picture, and neither is the threshold itself: reducing σ (a model that maps paraphrases closer together) and reducing ρ (a model that pushes lookalikes apart, which is what fine-tuning on hard negatives does). The lab exposes exactly these two through its embedding model selector.

Two practical consequences follow. First, a threshold is not portable: 0.90 on one embedding model is a different operating point from 0.90 on another, and changing the model without recalibrating silently changes your error rate. Second, libraries disagree on units. Some take a similarity (higher is stricter), others take a cosine distance equal to one minus the similarity (lower is stricter), and some vector stores return inner product or L2 distance depending on index configuration. A distance threshold of 0.1 is a similarity threshold of 0.9. Check which one your library version expects before copying a number from a blog post.

Hit rate and wrong-answer rate move together

The economics are simple to write down. Let h be the hit rate, c_llm the cost of a model call, c_e the cost of embedding and lookup, and π the precision of hits (the share of served hits that were actually correct). Then per request:

cost      = c_e + (1 - h) * c_llm
wrong     = h * (1 - π)                  # wrong answers per request served
saving    = h * c_llm - c_e

# lab defaults, illustrative: c_llm about $0.004, c_e about $0.00001
h = 0.50, π = 0.948  -> cost $1.99 per 1k, 26 wrong answers per 1k
h = 0.34, π = 0.993  -> cost $2.65 per 1k, 2.5 wrong answers per 1k

Both lines above come from the lab with the large general model, at thresholds 0.85 and 0.93. Tightening the threshold costs about $0.66 per thousand requests and removes roughly 23 wrong answers per thousand. Whether that is a good trade depends on what a wrong answer costs you, which is a product question, not an infrastructure one. For a store FAQ, a wrong return policy answer might cost a support ticket; for a medical or financial assistant the right number may be "no semantic reuse at all".

The last row is the point of the table. The tuned model reaches the loose threshold's hit rate at the strict threshold's error rate, because it changed the geometry instead of moving the line. When teams report that "semantic caching does not work for us", they have usually explored only the threshold axis.

Calibrating the threshold with labelled pairs

A threshold should be chosen from data, not intuition. The procedure that holds up in review:

1. Run the cache in shadow mode: compute the nearest neighbour and its similarity for every request, but always call the LLM. Log the request, the neighbour and the score.
2. Sample pairs across the similarity range, oversampling the band you are likely to choose (say 0.85 to 0.97), and label whether the cached response would have been an acceptable answer to the new request. An LLM judge can pre-label, but spot-check it by hand, because judges are lenient about exactly the subtle differences (negation, product variant, date) that cause harm.
3. For each candidate threshold compute precision of the would-be hits and the hit rate, then pick the lowest threshold whose precision meets the target with a confidence bound, not a point estimate.

The bound matters because the error rates you care about are small. If you label 300 would-be hits above a threshold and find zero bad ones, the "rule of three" gives a 95% upper bound of about 3/300, or 1%. To claim a wrong-hit rate under 0.3% you need on the order of 1,000 labelled hits with no failures. Teams that label 50 pairs and declare the cache "100% accurate" have learned almost nothing. Recalibrate whenever the embedding model, the normalisation step or the traffic mix changes, and re-sample a few hundred live hits every month.

Scoping: what the cache key must include

Similarity of the question text is necessary but not sufficient. Two identical questions can still need different answers if anything else in the prompt differs. Typical things that must be part of the cache scope, enforced as an exact-match metadata filter alongside the vector search:

• Tenant and permissions. An answer generated from documents one user can see must never be served to a user who cannot. This is a data-leak bug, not an accuracy bug.
• Model, system prompt and tool versions. When the prompt template changes, old answers reflect old instructions.
• Locale and channel. Currency, language and formatting rules differ.
• Conversation state. "What about the paperback?" means nothing without the previous turn. Embedding only the last message makes every follow-up collide with every other follow-up; either include a summary of the context in the key or skip the cache for multi-turn requests.

Do not cache personalised answers (order status, account balance) at all, and skip the cache when the request uses tools whose outputs change. A simple allowlist of cacheable intents, decided by a cheap classifier or by route, removes most of these risks before similarity is even consulted.

Staleness: TTL versus event-driven invalidation

A cached answer is a snapshot of the world when it was generated. When prices, policies or stock change, every entry that mentioned them becomes a confident wrong answer, and nothing in the similarity score reveals it. There are two tools. A TTL bounds how long any entry can live; it is cheap and needs no knowledge of the content, but it throws away good entries along with stale ones and still serves stale answers until expiry. Event-driven invalidation deletes the entries that depend on the changed data; it keeps the hit rate but requires you to record which source documents, products or policy versions each entry depends on.

The lab makes the trade visible. With the catalog price update event, 30% of intents change their correct answer halfway through the run. With no expiry, wrong answers jump to roughly 70 per thousand. A 30-minute TTL brings that down to about 7.5 per thousand but the hit rate falls from 34% to 22%. Invalidating only the touched entries keeps a 33% hit rate at about 3.5 wrong per thousand. In production the dependency record is usually the retrieval context: if the answer was generated from chunks with IDs and versions, store those with the entry and invalidate when any of them changes. RedisVL and GPTCache both support per-entry TTLs; dependency tracking is typically yours to build, so check what your version offers.

Capacity, eviction and the long tail

Query traffic is heavy-tailed. A few hundred questions account for a large share of volume, and a long tail of one-off questions never repeats. Every miss inserts an entry, so the one-offs churn the cache: with least-recently-used eviction and a small capacity, popular entries are evicted by traffic that will never hit. In the lab, with long-tail traffic on, growing capacity from 100 to 400 entries lifts the hit rate from about 17% to 24%, and each further doubling buys less because the extra room mostly stores one-offs. Memory is rarely the binding constraint for a semantic cache (a million 768-dimension float32 vectors is about 3 GB before index overhead), but lookup latency and index maintenance grow with size, and every stored entry is one more candidate for a wrong match. Admission policies help: only cache responses for allowlisted intents, or only after a question has been seen twice.

Lookup latency is the other budget. Embedding a short query and searching an approximate index typically costs a few to a few tens of milliseconds, against hundreds of milliseconds to seconds for generation, so a miss adds little. A cross-encoder or small judge model that verifies a candidate hit before serving it adds tens of milliseconds and catches many lookalikes; it is a reasonable middle ground when you cannot fine-tune the embedding model.

Knobs in real systems

Parameter names and defaults change between releases, so treat this table as a map of concepts and check the documentation for the exact version you deploy. Whatever the library, log the similarity score with every hit; without it you cannot audit or recalibrate later.

Worked example, failure modes and the lab

Suppose the hypothetical Belgavi's Book Store assistant serves one million questions a month at about $0.004 per model call, or $4,000 a month without a cache. A loose setup with 50% hits saves about $2,000 but, at the lab's precision, serves around 26,000 wrong answers a month. The 0.93 setup saves about $1,350 and serves around 2,500. Paying roughly $660 a month to avoid 23,500 wrong answers works out at under three cents per avoided error, which is an easy decision for most product owners once it is framed that way. The domain-tuned model would recover most of the savings; the cost of fine-tuning and labelling pairs is the real price of that option. All figures are calculated from the synthetic model, not measured.

The failure modes to monitor are predictable. Lookalike collisions show up as higher thumbs-down, regenerate or escalation rates on cached responses than on fresh ones; compare the two populations every week. Staleness shows up as complaints clustered after data changes; alert on hit traffic for entries older than your last catalog update. Scope leaks rarely show up in metrics at all, so test them explicitly with paired requests from different tenants. Silent drift happens when someone upgrades the embedding model and keeps the old threshold. Common mistakes: caching multi-turn follow-ups by their last message, sharing a cache across prompt versions, choosing a threshold from a handful of examples, and reporting hit rate on the dashboard without hit precision next to it.

The lab's missions follow that order. Mission one starts from a loose threshold and asks you to cut wrong answers while keeping 30% of traffic cached. Mission two turns on the catalog price update and asks you to stay correct without sacrificing the hit rate, which TTL alone cannot do. Mission three floods the stream with lookalike questions and adds a cost ceiling, so raising the threshold is no longer enough. Mission four squeezes capacity under long-tail traffic and rewards the smallest cache that still holds the popular questions.

Run the lab: Semantic cache: tune the threshold before it answers the wrong question

The lab below streams 2,000 synthetic questions over four simulated hours through a real nearest-neighbour cache. The top strip shows requests flowing into the cache and leaving as correct hits (green), wrong hits (red) or full LLM calls (blue). The histogram shows, for every request, the similarity of its nearest cached entry, coloured by whether reusing it would have been right; the dashed line is your threshold, and everything to its right is served from cache. Start by dragging the threshold across the red bars, then work through the missions.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Log the nearest-neighbour similarity score with every cache hit and miss, and put hit precision next to hit rate on the dashboard.
2. Run the cache in shadow mode for a week, label a few hundred would-be hits across the 0.85 to 0.97 band, and pick the threshold from a confidence bound.
3. Confirm whether your library takes a similarity or a distance threshold in the version you run, and write the unit next to the value in config.
4. Add tenant, permission scope, locale, model and prompt-template version as exact-match filters on every cache lookup.
5. Store the source document or product IDs each cached answer depends on, and invalidate those entries when the data changes.
6. Compare thumbs-down or regenerate rates for cached versus fresh answers weekly, and recalibrate after any embedding model change.

Sources and further reading

• GPTCache: a library for creating semantic caches for LLM queries (Zilliz, GitHub)
• RedisVL: Redis vector library with SemanticCache (GitHub)
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• The Faiss library

Related in this series:

• Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests
• Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
• HNSW Tuning for RAG: M, efSearch and the Recall Versus Latency Trade

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.