Evidence label: deterministic synthetic model. This page runs a small JavaScript model of prefix caching on a fleet of replicas. Synthetic chat sessions pick one of several system prompts and send several turns, each prompt containing the system prompt and the full history. Each replica keeps a radix tree over these prompt segments with LRU leaf eviction and a token capacity; a router picks the replica; prefill time depends only on tokens the tree did not already hold. Nothing here runs a model or measures a GPU; all numbers are calculated estimates.
Play the lab
How to play
Turn the cache on or off, set the prefix memory per replica, the number of replicas, the router, the spill threshold for cache-aware routing, where dynamic text sits in the prompt, and the request rate. Every change re-runs the same seeded traffic, so differences come from your change. Inject a timestamp in the prompt, a tenant surge or a burst at any time. Missions unlock in order; Load mission setup starts from the broken configuration, and stars reward fewer replicas or lower latency.
What the model assumes
Sessions arrive as a Poisson process; each has one or more turns (four on average) spaced 2 to about 10 seconds apart. System prompts are 1,500 tokens (200 before the point where a timestamp would be inserted), chosen with a Zipf skew across 6 prompts, or 40 during a tenant surge. User turns are 40 to 240 tokens and answers average 220. A replica prefills 10,000 tokens per second plus 15 ms per request, serves requests first come first served, and inserts each prompt and its answer into its tree. One GB holds 8,192 tokens of KV. Cache-aware routing picks the longest match unless that replica's backlog exceeds the least-loaded one by more than the threshold.
What it cannot tell you
Decode is not modelled, so the cache never competes with running sequences for memory, which in a real server shrinks it at peak. Real trees work on tokens rather than segments, real routers see an approximate view of each tree, and real traffic has structure beyond chats. Use the lab to understand the shape of the trade-offs, then measure hit rate and TTFT on your own traffic.
Read the full explanation
- Prefix Caching With Radix Trees: Reusing Prompt Work Across Requests (companion article)
- PagedAttention Block Tables: How vLLM Ends KV Cache Fragmentation
- Semantic Caching for LLMs: Similarity Thresholds and the Wrong-Answer Risk
All AI in Production labs · Lab catalog
- SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)
- vLLM documentation: engine arguments
- SGLang documentation
Views are my own and do not represent any employer, past or present.