Why it matters
KV compression extends context. Understanding shapes memory optimization.
Advertisement
The architecture
Quantize KV: int8, int4.
Evict: token importance selection.
Merge: similar tokens.
Advertisement
How it works end to end
Quantization: 2-4x memory.
Eviction: H2O keeps heavy hitters.
SnapKV: recent + important.
Sparse attention.