Why architecture matters here

Serving decisions collapse to KV cache math: how many sessions fit? Architecture matters because layers, GQA, dtype, and paging decide the answer.

Advertisement

The architecture: every piece explained

The top strip is the factors. L layers in model. H heads × d_h. N tokens — prompt + generated. Dtype bytes — bf16=2, int8=1, int4=0.5.

The middle row is refinements. Multiplier 2 for K + V. GQA scale — H_kv often 1/8 of H. Total per session bytes. Serving fit — sessions per GPU HBM.

The lower rows are ops. Paging — block-based like vLLM. Metrics — cache utilization. Ops — sizing budget + eviction.

KV cache math — layer × head × dim × dtype × tokens × 2 (K+V)size the biggest memory pool in LLM servingL layersconfigH heads × d_hper layerN tokenssequence + generatedDtype bytesbf16=2, int8=1, int4=0.5Multiplier 2K + VGQA scaleH_kv < HTotal per sessionbytesServing fitsessions per GPUPagingblock-based allocationMetricscache utilizationOps — sizing budget + eviction + capacitydoublereducesumcappagewatchwatchoperateoperate
KV cache sizing formula and paging.
Advertisement

End-to-end flow

End-to-end: 70B Llama-family with GQA 8:1, 80 layers, d_h=128. At 8k context, bf16 KV: 80×2×8×128×8192×2 = 2.7GB/session. On H100 80GB with 35GB for weights: 45GB free / 2.7GB = ~16 sessions. With int8 KV: 32 sessions. Paging avoids fragmentation.