Why it matters

Disk offload enables persistent sessions. Understanding shapes design.

Advertisement

The architecture

Ultra-cold cache on NVMe.

Serialize + compress.

Restore on session resume.

KV cache tiersActiveGPU HBMWarmCPU RAMColdNVMe / SSD3-tier tiering; NVMe serialization + compression to fit more sessions
KV disk offload.
Advertisement

How it works end to end

Serialization: quantize + compress.

Load: NVMe -> CPU -> GPU.

Persistent session use case.