A canary rollout rests on one assumption: if the new thing is bad, you will know before very many people are hurt. For ordinary software that holds. Errors spike, a pod crashloops, p99 latency doubles, and the abort fires in minutes on signals nobody has to interpret. Ship a new model checkpoint and that assumption quietly breaks. The failure mode you actually fear — “the answers got worse” — is subjective, arrives days late, and does not page anyone. Canarying an LLM is therefore a different design problem, one in which fast objective proxies carry the abort decision, slow subjective signals gate promotion, and the whole thing runs on GPUs that are far too expensive to duplicate casually. This is the mechanism: where the split lives, how traffic is assigned, what you gate on, and when you pull the cord.
Why shipping a model is not shipping a service
Progressive delivery works because detection time is shorter than blast radius. A 5% canary that regresses is caught in the first hundred requests. Model regressions are not shaped like that. A checkpoint that has become more sycophantic, worse at following a system prompt, or subtly more likely to hallucinate a citation produces perfectly valid HTTP 200s at normal latency. Every dashboard stays green while the product gets worse.
So the honest framing is that an LLM canary has two clocks. The fast clock measures the things that break like software: availability, latency, malformed output, tool-call failures. The slow clock measures whether responses are actually better, and it runs in days. The design job is to make the fast clock as sensitive as it can possibly be, and to never pretend the slow clock is running faster than it is.
Three kinds of change, three different risk shapes
“Deploying an LLM change” hides three very different rollouts. A weights change — a new checkpoint, a new fine-tune, a different quantization — alters behaviour everywhere at once and needs a second full set of GPUs to serve it. An engine change — a new vLLM or TensorRT-LLM version, a different attention backend, a fused kernel — keeps the weights but rewrites the arithmetic and the scheduling, so it moves throughput and latency hard and can shift outputs numerically. A config change — sampling parameters, max tokens, the system prompt, a tool schema, batching knobs — is cheap to revert and usually needs no extra capacity at all.
Roll them out separately. Bundling a checkpoint swap with an engine bump destroys attribution: when the canary degrades you cannot tell which half did it, and the safe response is to revert both.
The split belongs in the routing tier
The variant decision has to live in the gateway or router in front of the serving pools, not inside a pod. This is forced by the engine case: an engine upgrade is the pod, so the unit you roll forward and back is an entire replica set with its own GPUs, its own weights, and its own runtime. The router holds a weight for each pool, picks one per request, and — this is the part teams forget — stamps the resolved variant onto the response, the access log, and every metric it emits.
Without that label nothing downstream is computable. You cannot compare per-arm latency, you cannot attribute a support ticket, and you cannot answer “which version produced this bad answer” a day later. Mesh-level weighting (Istio, Argo Rollouts) splits connections; a flag or router layer splits requests, keyed on identity. For LLM traffic you want the latter.
Sticky assignment — nobody flips mid-conversation
Assignment must be a pure function of a stable key: hash the conversation or session id into a unit interval and send it to the canary when the hash falls below the current ramp fraction. Because the fraction only grows, widening 1% to 5% adds users without re-rolling the ones already assigned, and the cohort stays a superset of itself.
Randomising per request instead produces two distinct failures. The visible one is behavioural: a multi-turn conversation answered alternately by two checkpoints changes voice, formatting, and tool-calling conventions between turns, and any JSON or function-call contract that differs between versions can break mid-flow. The invisible one is a performance artifact. Conversations that keep landing on the same replica reuse their prefix KV cache; scattering turns across variants forces prefill to be redone and inflates time-to-first-token on both arms. You will read that as a canary regression when it is your own routing.
The signals that move fast enough to gate on
These metrics can carry an automatic abort: objective, cheap, and readable within minutes at modest traffic.
| Signal | Catches |
|---|---|
| TTFT and inter-token latency, kept separate | Scheduling and prefill regressions a blended p95 hides |
| Error, timeout and empty-response rate | Hard breakage, OOM, deadlocks in a new engine |
| Output-length distribution | Runaway generation, premature EOS, a broken stop condition |
| Tool-call and JSON parse failure rate | The sharpest fully objective quality proxy there is |
| Refusal rate | An over-aligned checkpoint declining ordinary work |
| Truncation rate and output tokens per request | Cost blowups and answers cut off at the limit |
None of these say the answers are good. All of them say, very quickly, that something changed — and a shift in the output-length distribution is usually the earliest cheap evidence that behaviour moved at all.
The signal that lags, and how to shift it left
Human preference is the metric you actually care about and the one you cannot gate on. Explicit feedback is sparse, arriving on a small minority of sessions and skewed toward people who are annoyed; at 5% traffic it takes days for a thumbs-up rate to move a confidence interval far enough to act on. Downstream measures — task completion, retention, escalation to a human — lag further still.
The practical bridge is sampled model-graded evaluation on live traffic: pull matched samples from both arms each hour, score them against a rubric with a judge model, and watch the gap. That is fast enough to promote or hold a stage, and far too noisy and expensive to fire an automatic rollback on. Keep the division clean: fast proxies abort, slow signals promote. A rollout stage that no fast signal condemns and no slow signal endorses should sit and bake, not advance.
Automated abort: triggers, thresholds, bake time
Encode two rule types. Absolute guardrails fire when the canary breaches an SLO outright. Relative rules fire when the canary is worse than the control arm by more than a set margin — and relative is what makes the system trustworthy, because comparing the canary against a concurrently running control cancels out diurnal load, a noisy neighbour, or an upstream provider having a bad morning. Comparing against yesterday’s baseline just fires every Monday.
Three details do most of the work. Require a minimum sample before any rule can trigger, or 1% traffic will abort your rollout on a single timeout. Give each stage a bake time covering at least one full traffic cycle, since load-dependent regressions only appear at peak. And make abort one-way: flip the weight to zero instantly, then require a human to restart. Auto-retry loops flap, and flapping is worse than being down on one version.
The GPU bill for running two versions at once
Here is where the analogy with a stateless web canary collapses. Five percent of web traffic can be served by five percent more pods. A model replica is indivisible: the weights must fit, and a tensor-parallel replica occupies its whole group of GPUs whether it is serving 1% of traffic or 40%. If a replica spans eight GPUs, the smallest canary you can physically deploy is eight GPUs. That, far more than tooling, is why teams skip canarying models.
It also biases the measurement. A canary pool at 1% traffic sits nearly empty, so it batches shallowly, never queues, and posts latency it cannot possibly hold at full load. Do not accept a latency verdict from an under-loaded arm: compare at matched concurrency, or ramp until batch sizes on both pools are comparable before you believe the numbers.
Engine and kernel upgrades — different bits, not wrong bits
Swap in a new attention kernel, a different GEMM, or a changed reduction order and floating-point results shift in the last bits. Propagate that through a softmax and a sampler and eventually a different token wins somewhere, after which the two completions diverge entirely. Nothing is broken. The outputs are simply not identical, and they were never going to be — most batched serving stacks are already non-deterministic run to run, because a request’s numerics depend on which other requests it was batched with.
Byte-diffing old versus new is therefore a false-positive generator, and teams that try it end up ignoring the alarm. Check distributions instead: length, refusal rate, tool-call success, and a judged sample-versus-sample comparison. Save exact-output determinism checks for a pre-production harness with greedy decoding and a fixed batch, where you control the variables. Quantization changes deserve a stricter bar than kernel changes — those alter the weights themselves, not just the order of arithmetic.
A ladder that survives contact
A workable sequence: internal and employee traffic first, then 1%, 5%, 25%, 50%, 100%, each stage sticky and each with an explicit bake time. The early stages exist to catch hard breakage on the fast clock and nothing else — they carry no statistical power for quality. The later stages are where sampled evaluation and any preference signal become readable, so hold longest at 25% and 50%, not at 1%.
Two rules keep the ladder honest. Keep the previous version warm and able to absorb 100% until the canary clears the final stage plus a soak period; a rollback that requires re-provisioning GPUs is not a rollback, it is an outage with extra steps. And change one variable per rollout. The temptation to bundle the new checkpoint with the engine upgrade while you are already paying for double capacity is exactly how a clean rollout becomes an unattributable one.