Prefix Caching — Reusing KV Cache Across Requests

Why it matters

Shared prompts common in production. Prefix caching cuts cost. Understanding shapes efficient serving.

The architecture

Identify common prefix.

Compute KV cache once; store.

New requests with same prefix reuse cache.

How it works end to end

vLLM automatic: detects prefixes via hash of tokens; reuses.

SGLang RadixAttention: tree structure of shared prefixes.

Cache eviction: LRU when memory tight.

Impact: 2-5x speedup for prefix-heavy workloads.