vLLM on GPU — Continuous Batching for High Throughput

Why it matters

vLLM is the default modern LLM serving. Understanding shapes deployment.

The architecture

PagedAttention: KV cache in fixed-size blocks. Virtual → physical block mapping. Eliminates fragmentation.

Continuous batching: add new requests when slots free.

How it works end to end

Prefix caching: shared prompts reuse KV cache.

Speculative decoding integration: drafter + verifier.

Multi-GPU: tensor parallel + pipeline parallel supported.

Quantization: AWQ, GPTQ, FP8 supported.