Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a semantic cache. It generates 1,200 question intents as random 16-dimension unit vectors with heavy-tailed popularity, creates paraphrases by adding noise, and pairs some intents with lookalike twins that need different answers. Each request is embedded, compared by cosine similarity against every live cache entry, and either served from cache or sent to a simulated LLM call that inserts a new entry. Nothing here calls a model or a vector database; all numbers are calculated estimates that make the trade-off visible.
Play the lab
How to play
Drag the similarity threshold and watch the dashed line move across the histogram. Teal bars are requests whose nearest cached entry had the same intent, so reuse would be correct; red bars are requests whose nearest entry belonged to a different intent; amber bars are same-intent entries that went stale after a catalog update. Everything to the right of the line is served from cache. Switch the embedding model to see the bars separate or overlap, and use the TTL, capacity and the two toggles to manage staleness and churn.
Every change re-runs the same synthetic stream, so differences come from your change; use New random seed to check that a conclusion survives different traffic. Missions unlock in order. Load a mission setup, fix it, and earn more stars for keeping the hit rate high or the cache small.
What the model assumes
Paraphrase noise and lookalike distance differ by embedding model: the small general model spreads paraphrases widely, the large one less, and the domain-tuned model also pushes lookalike pairs further apart, as fine-tuning on hard negatives aims to do. A quarter of requests are one-off questions (55% with the long-tail event). A model call costs about $0.004 and 0.7 to 3 seconds; a cache hit costs an embedding plus lookup. The cross-encoder check catches 85% of wrong-intent hits and wrongly rejects 3% of correct ones. Eviction is least recently used, and expired or invalidated entries are reused first.
What it cannot tell you
Real embeddings are not Gaussian clouds, real lookalikes are not neatly paired, and real wrong answers vary in harm. The model ignores approximate-search errors, multi-turn context, partial staleness and the cost of labelling or fine-tuning. Use it to understand why the threshold alone cannot fix lookalike collisions or stale data, then calibrate your own threshold on labelled pairs from your own traffic.
Read the full explanation
- Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk (companion article)
- Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests
- Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
All AI in Production labs · Lab catalog
- GPTCache: a library for creating semantic caches for LLM queries (Zilliz, GitHub)
- RedisVL: Redis vector library with SemanticCache (GitHub)
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
- The Faiss library
Views are my own and do not represent any employer, past or present.