Almost everything written about LLM guardrails is about policy: what should be blocked, which taxonomy to use, who signs off. This article is about the other half. A guardrail is a serving component: one or two extra model calls on the hot path of every request. It consumes GPU time you were spending on the product model, it can time out, it can be wrong in two directions, and it must judge a streaming response before the user has finished reading the tokens you already sent. Treat it as an inference workload with its own latency budget, capacity plan, and failure mode, or it becomes the slowest and least reliable box in your serving diagram.
A classifier in the request path, not a document
Strip away the taxonomy and a guardrail is a classifier: text in, a label and a score out. In production it is usually a small language model or a fine-tuned encoder — Llama Guard and its relatives, or a framework such as NeMo Guardrails or Guardrails AI orchestrating several checks — plus deterministic rules (regex denylists, schema validation, URL allowlists) that cost nothing and should always run first.
The systems consequence is immediate. Every guarded request issues two or three inference calls instead of one, and they are serialized: you cannot start generating until the input check clears, and you cannot release the answer until the output check clears. Serialization is what makes this an architecture problem rather than a configuration problem. Two adjacent concerns are out of scope: prompt injection is adversarial-input work with its own article, and PII detection is a data-handling problem with another. What follows is guardrails as a latency and availability problem.
Placement — before the model, after it, or both
Three positions, catching different things. An input guard reads the user turn before generation; it is the cheapest place to stop something, because refusing there costs one small forward pass and zero tokens from the expensive model. An output guard reads what the model produced, and it is the only place that catches a harmful completion, since a benign-looking prompt can still produce a bad answer. Running both is the common choice.
The asymmetry worth internalizing: the input guard is a cost saver as much as a safety control, killing doomed requests before they occupy KV cache and decode slots. The output guard is pure added latency, running after you have already paid for generation.
The latency budget each placement spends
Price the placements against the two numbers users feel. An input guard lands squarely inside time to first token: its full duration is added before the product model even starts prefill, and it is unhideable. An output guard on a non-streaming endpoint adds to end-to-end latency only, which users tolerate better because they are already waiting.
A guard pass over a short prompt on a datacenter GPU is typically a small fraction of the product model’s own prefill — the guard is one or two orders of magnitude smaller — but that ratio collapses when the prompt is long, because the guard must read the whole thing too. A 32k-token context means a 32k-token guard prefill. Measure rather than assume, and write the guard’s share into the SLO explicitly: it may spend at most a stated slice of the TTFT target, and it pages when it exceeds that slice, instead of showing up as an unexplained p99 regression.
Guard calls are prefill-heavy and decode-free
This is the observation that makes guard capacity easy to reason about. A guard call reads a long input and emits almost nothing — a label, a score, maybe a handful of tokens of justification. It is essentially all prefill, and prefill is compute-bound: it saturates tensor cores with large matrix multiplies and scales with input length.
The product model’s decode phase is the opposite: memory-bandwidth-bound, one token at a time, starved for arithmetic intensity. A guard workload and a decode workload want different things from the same silicon, which is why mixing them naively hurts. It also means guard throughput responds well to batching — more independent prefills fill the machine — and barely at all to the tricks that help decode. And because the guard emits so few tokens it holds KV cache only briefly, so a guard pool is sized by compute and arrival rate, not by the cache-capacity arithmetic that dominates product-model sizing.
Same GPU or a separate pool
Colocating the guard with the product model is tempting: no network hop, no second deployment, and a small guard’s weights fit in the slack HBM. The costs are real. Those weights come out of the memory you were giving to KV cache, so maximum concurrency drops. Worse, a guard prefill injected into a running decode batch is a compute-heavy interruption: every in-flight decode stalls behind it, spiking inter-token latency for users who have nothing to do with the guarded request.
A separate pool converts that interference into a network round trip, a sub-millisecond hop plus RPC overhead, and buys independent scaling — guard and product traffic do not grow at the same rate, and a guard upgrade no longer means redeploying the serving fleet. You pay with a second fleet to keep warm and a new failure domain. The middle path is hardware partitioning: a MIG slice or a dedicated stream priority, so the guard shares a device without sharing a scheduling queue.
Validating a stream you have already sent
Streaming breaks the clean before/after model. If tokens go to the browser as produced, an output guard that runs on the finished answer is useless: by the time it fires, the user has read the text. Three honest options.
Hold-back: buffer the last N tokens and release them only once a check has passed, so the visible stream lags generation by a fixed window. That costs a constant amount of perceived latency and nothing else, and it is the default worth starting with. Chunked scanning: run the guard every k tokens or at sentence boundaries on the growing prefix. Done naively this is quadratic — each scan re-reads everything before it — so it is only viable if the guard reuses the KV cache of the previous prefix, the same prefix-sharing mechanism the product model uses. Retraction: stop the stream and replace the message client-side: cheapest to build, worst experience, and you cannot unsend what was screenshotted.
Batching guard calls and caching verdicts
The guard is a model server, so all the usual throughput levers apply. Because guard requests are short and independent, they batch extremely well: a small queue with a few milliseconds of admission delay can lift guard throughput sharply at a latency cost that disappears inside the product model’s own prefill. Continuous batching helps less than usual here, since there is barely any decode to interleave — what you want is a large prefill batch.
Caching is the other lever. Key a verdict on a hash of the normalized text plus the guard model version plus the policy version, so a policy change invalidates the cache instead of silently serving stale decisions. Hit rates are high wherever traffic is templated: system prompts, retrieved documents, tool descriptions, repeated user phrasings. Cache on exact normalized text only — fuzzy or embedding-similarity matching invites a near-collision that inherits an approving verdict it never earned.
Fail-open, fail-closed, and the timeout that decides
Every guard eventually has a bad minute. What happens then is a design decision you should make on purpose. Fail-open serves the request unguarded: availability is preserved and the safety control silently disappears for the duration of the incident. Fail-closed refuses: the control holds and your product is down. Neither is universally right; the answer is per-surface — fail-closed on a public, unauthenticated, high-risk surface, fail-open on an internal tool where the guard is defence in depth behind other controls.
The parameter that actually decides your behaviour is the timeout, and it must be set from the latency you have left in the SLO, not merely shorter than infinity. A hanging guard is far worse than an erroring one: it consumes the whole budget before failing anyway. Put a circuit breaker in front of it so repeated failures stop being retried, and give the breaker a cheap deterministic fallback so degraded is not the same as absent.
False positives are the cost nobody measures
Guard tuning has an asymmetry that quietly wrecks products. A false negative is loud: someone reports it, it becomes an incident, the threshold gets tightened. A false positive is silent: the user sees a refusal for a legitimate request, assumes the product is useless, and leaves. Nothing in your logs looks like an error. Left alone, that asymmetry ratchets the guard tighter until the product no longer answers ordinary questions.
The countermeasure is measurement. Track block rate as a first-class metric, split by policy category, surface, and user segment, and treat a step change after a deploy as a regression. Sample blocked requests for human review to get a real precision estimate rather than a vendor-supplied one. Set thresholds per category rather than globally. And watch session abandonment after a refusal — that is the number that turns an invisible cost into a visible one.
Shipping a policy change without shipping an outage
Guard configuration changes far more often than model code, usually under time pressure after an incident, and it changes what every user is allowed to do. That combination deserves the release discipline you give a binary. Run new policies in shadow mode first: the guard executes and logs its verdict, but enforcement stays on the old rules, so you see the block-rate delta on live traffic before anyone is refused. Keep a replay corpus of known-bad and known-good requests and diff verdicts against it on every change.
Then roll out by traffic percentage with block rate as the canary signal, and keep the policy version in the request log and the verdict cache key so decisions are attributable and rollback is instant. Finally, give the guard its own SLO — latency, availability, block rate — and page on it. A safety component nobody is on call for is one you will discover has been failing open for a week.