KV Cache Offload to CPU

Why it matters

CPU offload extends effective GPU capacity. Understanding shapes long-context serving.

The architecture

Hot cache on GPU.

Cold cache on CPU RAM.

Fetch back on reactivation.

How it works end to end

Session detection: LRU idle.

Transfer via PCIe.

Reactivation cost: milliseconds.

vLLM + SGLang support.