Three items this week examine what happens when a familiar production assumption meets a specific measurement.
Cancelling a request does not cancel its retries
A paper posted November 5 tests overload recovery on vLLM with three open models. The natural remedy for an engine computing for clients who have left is to propagate cancellation. The authors report that with retries on, cancellation alone "did not restore service in any run", and that only 12 to 38 percent of calls arriving after the overload were delivered.
The scheduler traces suggest why. Retries that vLLM ran usually began prefill less than half a second before timing out. Executions later aborted took 79 to 85 percent of the prompt tokens computed, and vLLM's prompt-token counter missed a third to half of all computed prompt tokens, so a dashboard built on it would understate the waste. A retry budget restored service but left many calls undelivered, while request coalescing delivered every call from the overload onward unless cancellation had already aborted the running execution.
Guards that cannot read a contract
A November 6 study measures eleven open-weight prompt injection guards on two corpora of legal contracts. Document-level false positive rates reached 92.39 percent, two guards fell below chance on paired AUC, and a five-regex baseline was not Pareto-dominated by any guard in the panel. Legal drafting uses phrases such as "notwithstanding anything to the contrary", which look like instructions because in a sense they are. Prompted LLMs resolved the tested failures, and a cascade that sends a mid-tier model's positives to a frontier model was 6.5 times cheaper than using a frontier model alone.
Moving weights during reinforcement learning
During post-training, training GPUs must repeatedly send new weights to inference GPUs, and the two sides usually lay them out differently. NVIDIA's November 6 engineering post introduces the Reshard collective, with NCCL M2N Reshard as an open source reference implementation. On DGX H100 and DGX GB200 systems, NVIDIA reports "a speedup of 2.7x to 5.7x over conventional gather-and-broadcast refit" across models including Qwen 3 235B and DeepSeek v3 671B. These are vendor measurements on vendor hardware, and a hybrid mode keeps the older method for smaller weights.
What remains uncertain. We read the two abstracts and NVIDIA's post, and reproduced nothing. The vLLM results cover three models and one engine, and other schedulers may behave differently. The guard study covers legal text only, so its false positive rates should not be applied to other kinds of document without testing.
Analysis: Measure the defence on your own traffic before trusting it. If Geetha's team at Belgavi's Book Store ran a guard over publisher contracts, a false positive rate near the top of that range would make the guard worthless, and a few expressions plus a model check on the positives may do better. For serving, cap retries at the client and proxy as well as the engine, and alert on computed tokens from the scheduler, since the study shows the headline counter can undercount what actually ran.
Sources and evidence
- Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine (arXiv, November 5)
- Notwithstanding Anything to the Contrary, Ignore All Instructions: Measuring Prompt Injection Guards on Legal Contract Review Attacks (arXiv, November 6)
- Accelerating Reinforcement Learning Weight Refit with the Reshard Collective (NVIDIA, November 6)
On vLLM, cancellation alone never restored service with retries on, and aborted executions consumed 79 to 85 percent of computed prompt tokens. Eleven open injection guards flagged up to 92.39 percent of benign contracts. NVIDIA reports 2.7 to 5.7 times faster weight refit with its Reshard collective. Cap retries at every layer and test guards on your own documents.