Why it matters

Serving optimizations can 5-10x throughput on the same hardware. For SLM deployment at scale, mastering them cuts cost proportionally.

Advertisement

The architecture

Batching: process multiple requests together. GEMM is much faster for batched matmul.

Continuous batching: don't wait for whole batch to finish; add new requests as slots free up.

Serving optimizationsBatchingmany at onceContinuous batchingadd/drop dynamicallySpeculative decodingsmall model draftsvLLM implements continuous batching + PagedAttention; standard for LLM serving
Serving techniques.
Advertisement

How it works end to end

Speculative decoding: small draft model generates candidate tokens; large model verifies in parallel. Cuts latency 2-3x.

Multi-query / grouped-query attention: reduces KV cache, enables larger batches.

Kernel fusion: fuse operations (attention + softmax + residual) to reduce memory traffic.

Speculative + batching + FlashAttention combined can produce 10x throughput improvements.