PagedAttention
KV cache paged like OS virtual memory. Non-contiguous storage. Massive memory efficiency. Enables batches 10x+ larger.
Advertisement
Continuous batching
New requests join active batch mid-generation. Old requests exit. Never waste GPU on padding. Vs static batching.
Advertisement
Compatibility
Llama, Qwen, Mistral, DeepSeek, most modern architectures. Quantization: AWQ, GPTQ, FP8, INT4.