vLLM 0.34.0 was released on November 16, 2026, as confirmed by its package record. The release notes enable extensible KV caching by default, changing when the serving engine decides how much memory to allocate.
Cache sizing moves after warmup
The new default measures available memory after graph capture and warmup. The implementation discussion, merged November 2 and cited here as background, describes reserving virtual addresses first and mapping physical memory later. Keeping addresses stable lets the allocation grow without relocating the addresses used by captured graphs.
The project's memory documentation separately identifies maximum context length and sequence count as memory controls, and notes that CUDA graphs consume GPU memory. Together, these sources suggest checking the complete serving configuration when comparing cache sizes.
A larger cache does not settle latency
The maintainers report MiniMax-M3 cache capacity on H200 increasing from 433,000 to 672,000 tokens. They also report 6-18% higher time per output token for some H200 models at concurrency 128. These are project-reported observations from specific configurations, not measurements reproduced by AICassindra or expected results for every workload.
The release provides an option to disable extensible caching. The tagged license is Apache-2.0; no project implementation code is reproduced here.
Compare capacity and response time together
Proposed synthetic experiment, not executed: prepare short and long prompts, then compare caching enabled and disabled at concurrency 1, 16 and 128. Keep the model, GPU, output-token cap and prompt set fixed. Record allocated cache, first-token latency, time per output token, aggregate throughput and failed requests.
Choose an acceptable response-time bound before inspecting the results. A configuration that serves more simultaneous sequences would still fail that requirement if individual responses became too slow. Publishing both measurements would let readers see the trade-off instead of treating the largest cache as the automatic winner.
Sources accessed November 17, 2026. Views are my own and do not represent any employer, past or present.
Sources and evidence
Evaluate cache capacity, throughput and response latency together; the release's H200 results do not establish a universal performance gain.