Shared KV Cache

Why it matters

Shared KV saves huge memory in chat services. Same system prompt shared across all users.

The architecture

Detect common prefix.

Reference same blocks.

Copy on write when diverges.

How it works end to end

Hash prefix; share blocks.

Copy-on-write on divergence.

vLLM prefix caching, SGLang RadixAttention.