Why it matters

Inference kernels dominate serving cost. FlashInfer is state-of-art. Understanding enables fast inference.

Advertisement

The architecture

Attention variants: prefill (multi-token, GEMM-heavy), decode (single-token, GEMV-heavy).

PagedAttention support for KV cache management.

FlashInfer kernelsPrefill kernelGEMM optimizedDecode kernelGEMV optimizedPagedAttentionblock-based KVDifferent kernels for different attention shapes; auto-select at runtime
Kernel selection.
Advertisement

How it works end to end

Cascade inference: prefix caching for shared prompt fragments.

Speculative decoding kernels: verify multiple candidates efficiently.

Integration: vLLM, SGLang use it.