vLLM 0.32.0: FP8 and NVFP4 KV Caches, the KV Memory Arithmetic, and the Breaking Changes to Check Before Upgrading

What shipped and why it matters

vLLM v0.32.0 was published on 18 October 2026. Its release notes count 814 commits from 363 contributors and group the highlights under structured decisions, low-precision KV caches, Model Runner V2, startup and serving speed, model support, security fixes and breaking changes. For a team running vLLM in production, two of those groups decide the upgrade plan. The KV cache changes alter how many tokens fit on a GPU, which is the main driver of concurrency and therefore of cost. The breaking changes alter behaviour that clients and deployment scripts depend on. This article covers both, with the arithmetic and the evidence the pull requests themselves provide.

Low-precision KV caches in this release

According to the release notes, an NVFP4 KV cache now runs on SM8x and SM12x GPUs (PR #46963), on SM90 (PR #60620), with XQA decode on SM12x, and on MiniMax-M3 sparse attention on SM100. GLM-5.x models default to an FP8 KV cache on SM100. ROCm gains a 4-bit KV cache type. PR #46963 explains the starting point: before it, --kv-cache-dtype nvfp4 had a paged write path only on SM100, so the data type could not be used on Ampere or consumer Blackwell parts at all. The PR also refuses two combinations at backend selection, before weights load, on those GPUs: attention with head size 64, where it reports misreads of 10 to 15 percent, and attention sinks.

The vLLM documentation on quantized KV caches describes the FP8 options. Scales can be per tensor or per attention head; per-head scales need the Flash Attention backend and calibration through llm-compressor. Without calibration, kv_cache_dtype="fp8" sets every scale to 1.0, and the documentation recommends calibrating with a dataset for the best accuracy. A --kv-cache-dtype-skip-layers flag keeps chosen layers, such as sliding-window attention layers, at the model's native precision.

The KV memory arithmetic

The KV cache stores one key vector and one value vector per token, per layer, per KV head. Its size per token is therefore 2 x layers x KV heads x head dimension x bytes per element. Take a hypothetical model with 32 layers, 8 KV heads and a head dimension of 128. In BF16, at 2 bytes per element, that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, 128 KiB per token. FP8 halves it to 64 KiB. A 4-bit format stores half a byte per element plus scale factors; if each block of 16 elements carries a one-byte scale, the cost is 4.5 bits per element and the total about 36 KiB per token. The exact overhead depends on the format's block layout, so treat the 4-bit row as an estimate.

Illustrative python, checked for syntax; adapt names and storage to your system.

def kv_bytes_per_token(layers, kv_heads, head_dim, bits_per_value, scale_bits_per_value=0.0):
    return 2 * layers * kv_heads * head_dim * (bits_per_value + scale_bits_per_value) / 8

budget = 40 * 2**30                       # bytes left for KV after weights and activations
for name, bits, scale in [("bf16", 16, 0), ("fp8", 8, 0), ("4-bit", 4, 8 / 16)]:
    per_token = kv_bytes_per_token(32, 8, 128, bits, scale)
    tokens = int(budget // per_token)
    print(name, int(per_token), tokens, tokens // 8192)

The pull request that made FP8 the default for GLM (PR #60140) reports the real-world version of the first two rows: with a budget of 458,880 tokens, KV memory per rank fell from 40.13 GiB to 20.69 GiB. In that PR's benchmark, at concurrency 1, 4 and 16 with 8,000-token prompts, throughput rose by 2.3 to 5.5 percent and median time to first token rose by 6.9 to 11.1 percent. The author accepted the TTFT cost for the memory and throughput. Whether you should depends on whether your service level is written in TTFT or in throughput.

Accuracy is a per-model question

Halving or quartering KV memory is only useful if answers stay correct. PR #46963 includes its author's own evaluations, which show why this must be measured per model. On GSM8K with all 1,319 questions, Qwen3-0.6B scored 0.384 with a BF16 KV cache and 0.174 with NVFP4. On a 100-question sample, Qwen3-4B, with 32 query and 8 KV heads, scored 0.920 and 0.910 in two BF16 runs and 0.880 in both NVFP4 runs. A long-context retrieval test on Qwen3.5-4B gave an overall match ratio of 0.8523 with the default cache and 0.8452 with NVFP4, with differences in both directions at individual needle counts. The PR states that SM86, SM89, SM120 and performance were not tested.

The reading is not that NVFP4 is bad or good. A small model lost more than half its accuracy on arithmetic reasoning, a larger one lost a few points, and long-context retrieval barely moved. Run your own task evaluation, at your context lengths, before enabling a 4-bit cache, and prefer calibrated FP8 scales over the uncalibrated default.

Breaking changes to check

Two rows deserve detail. The transformers pin collides with the calendar: transformers 5.20.0 was released on 15 October and 5.20.1 and 5.20.2 followed by 19 October, so an environment that upgrades everything at once will either refuse to resolve or quietly downgrade transformers. Pin both packages together.

The max_tokens change fixes a long-standing mismatch with how clients use the parameter. PR #57035 explains that the input budget used to be computed as the context length minus the requested max_tokens, so a request was rejected when prompt plus max_tokens exceeded the context. Its example: a 128,000-token context, a 62,466-token prompt and max_tokens of 65,535 used to be rejected; now the prompt is accepted and generation is clamped to 65,534 tokens. Requests that used to fail fast will now run, and may stop at the context limit instead.

The CORS change, PR #61009, replaces a default allowed-origin list of a single wildcard with an empty list, so browser pages on other origins can no longer read responses from a reachable server that has no API key. Deployments that serve a browser front end must name its origin.

An upgrade sequence

1. Build a new image on Python 3.11 or later with vllm 0.32.0 and the pinned transformers and tokenizers versions; do not let a dependency bot touch either separately.
2. Diff your launch flags against the removals and the new CORS default; add --allowed-origins only where a browser really calls the server.
3. Replay a sample of production requests and compare responses, paying attention to requests near the context limit, which now behave differently.
4. Measure KV capacity, throughput and TTFT at your real concurrency with the current KV dtype; only then try FP8, and run your task evaluation with it.
5. Treat NVFP4 KV as an experiment per model: evaluate accuracy at your context lengths, and expect a refusal at startup for head size 64 or attention sinks on SM8x and SM12x.

Failure modes

• Reading the release-note throughput percentages as your expected gain; they are each PR’s benchmark on specific models and GPUs.
• Enabling uncalibrated FP8 or NVFP4 KV without a task evaluation, and discovering the accuracy loss from users.
• Upgrading transformers to 5.20 alongside vLLM 0.32.0 and getting a resolver failure or a silent downgrade.
• A browser dashboard that stops working after the CORS change, fixed by setting the wildcard back instead of naming its origin.
• Clients that depended on fast rejection when prompt plus max_tokens exceeded the context, now receiving long truncated generations.

Practice and related reading

Further reading:

• KV cache size in depth
• KV cache reuse with shared prefixes
• Prefill and decode disaggregation
• Speculative decoding

Sources and scope

Checked against these sources on 20 October 2026. Event: vLLM v0.32.0 release, 18 October 2026; recheck it before acting. Examples and figures are illustrative, not measurements.

• vllm-project/vllm release v0.32.0, 2026-10-18
• vLLM v0.32.0 requirements/common.txt (transformers and tokenizers pins)
• vLLM documentation: Quantized KV Cache
• vllm-project/vllm PR #60140: FP8 KV cache by default for GLM
• vllm-project/vllm PR #46963: NVFP4 KV cache on SM8x and SM12x with FlashInfer
• vllm-project/vllm PR #57035: treat max_tokens as an output upper bound, not an input reservation
• vllm-project/vllm PR #61009: restrict default CORS origins