Why AI Applications Need Safety Models Beyond Built-In Refusals

Safety model engineering / Guide 1 of 5. A source-linked decision guide with synthetic technical examples. No model benchmarks were run for this series.

The generator and the application have different responsibilities

A model that usually refuses harmful requests still does not know every rule of the application around it. A generator predicts an answer within its training and supplied instructions. An application must also decide which inputs it accepts, which resources a principal can access, what content it releases, and which side effects it permits. Those decisions remain necessary when the generator is well aligned. A separate safety classifier creates an explicit place to test a particular policy rather than hoping the generator follows it on every turn.

Consider a synthetic fixture assistant with permission to read /fixtures/report.txt. The user asks for a summary. A retrieved paragraph contains an instruction to overwrite /fixtures/result.txt. Neither sentence needs to contain violent or otherwise prohibited content. The failure is a change in authority: document text has become an instruction. A content moderator alone cannot establish whether the write was authorized. The application needs source separation and a permission check as well as any model-based screening.

Google recommends input and output safeguards because careful prompting and safety tuning do not eliminate harmful outputs. Its guidance also warns that excessive filtering damages utility. The engineering question is therefore which residual risk a classifier reduces, at what false-positive cost, and how failures are contained. It is not whether an extra model makes the system safe in the abstract. Google: application safeguards.

Safety is a specialization, while LLM and SLM describe scale

The phrase safety model describes a job. LLM and SLM describe broad model-size categories whose boundaries are not standardized. These categories overlap. ShieldGemma text models are fine-tuned decoder models derived from Gemma 2, with 2B, 9B, and 27B variants. They are language models specialized for policy classification. Prompt Guard 2 instead uses compact classifier variants with 22M and 86M parameters. Both can be called safety models, but they expose different capabilities and operational contracts. Google: ShieldGemma text model card; Meta: Prompt Guard 2 86M model card.

A general language model can also be prompted to judge safety. That makes it a useful baseline, especially when policy interpretation is complex. A dedicated classifier can offer a narrower output format, a clearer training objective, or a smaller deployment footprint. Those are reasons to evaluate it, not proof that it outperforms a larger model. The right comparison holds the policy, language, input set, and error costs constant.

Avoid describing a safety model as a second intelligent supervisor that understands everything the first model missed. It is another fallible inference component. When both models share training biases, unsupported languages, or misleading context, their errors can correlate. Independence of deployment is useful for release management; it does not demonstrate statistical independence of mistakes.

Four questions that should never share one unlabeled score

A content verdict cannot substitute for an authorization decision.
Decision	Question	Required context
Content policy	Does the content violate a named harm policy?	Content, modality, policy definition
Prompt attack	Does untrusted text attempt to redirect behavior?	Source boundaries and attack scope
Grounding	Does the supplied evidence support the answer?	Answer plus authorized references
Authorization	May this principal perform this operation?	Identity, resource, operation, approval state

A quoted attack in a security explanation can look malicious to an injection classifier while remaining legitimate material for read-only analysis. Conversely, an innocuous instruction to delete a fixture can be unauthorized without violating a content-harm taxonomy. Store the type of decision with every verdict so downstream code cannot mistake one for another.

Grounding is different again. An answer can be harmless but unsupported by its references. Granite Guardian includes criteria for grounding and function-call hallucinations, which demonstrates that safety and quality checks can share a model family without becoming the same task. Its yes/no output must be interpreted according to the selected criterion; a yes for groundedness does not have the same meaning as a yes for harmfulness. IBM: Granite Guardian 4.1 8B model card.

When a dedicated check earns its operational cost

A dedicated check is useful when a consequential policy boundary can be stated, tested, and enforced. Examples include screening generated text before release, scanning retrieved chunks for instruction attacks, or rejecting unsupported image categories before routing to a suitable service. Separate deployment lets a team revise a classifier or threshold without replacing the generator. It also makes disagreements visible instead of burying them in the generator's response style.

The check must have an owner, a defined input contract, and an action for every outcome. Safe, unsafe, unsupported, timeout, and malformed output are different states. If the parser turns every unrecognized response into allow, the guard is weakest when its dependency fails. If every timeout becomes a permanent refusal, availability and usability can collapse. Define a bounded retry or review path and an appropriate degraded mode for the specific boundary.

Do not add a model when deterministic code answers the question exactly. A path allowlist, maximum payload size, JSON schema, or identity check does not need probabilistic judgment. A model is more appropriate for ambiguous language and contextual policy interpretation. Keep exact controls exact, and use the classifier to cover a clearly identified uncertainty rather than to replace basic validation.

A minimal contract makes failures observable

# Synthetic application contract, not a vendor response schema.
from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class Verdict:
    task: str
    status: Literal["allow", "block", "review", "unsupported", "error"]
    policy_version: str
    model_revision: str
    reason: str

def release_answer(verdict: Verdict) -> bool:
    return verdict.task == "output-content" and verdict.status == "allow"

assert not release_answer(Verdict("input-attack", "allow", "v1", "r1", "none"))
assert not release_answer(Verdict("output-content", "error", "v1", "r1", "timeout"))

This local example deliberately separates a classification result from a release decision. The adapter for a particular provider must map its outputs into this contract using that model's documented labels and supported categories. An absent category is not a negative detection. A probability-like score is not necessarily calibrated risk. Record the policy and revision so a later disagreement can be reproduced.

Operational logs should minimize stored content. Reason codes, timing, category, revision, and decision are often enough for aggregate monitoring. Retaining complete prompts and tool outputs can create a second sensitive dataset. Synthetic evaluation fixtures help debug the decision path without copying personal data into reports.

Safety checks complement the boundaries that stop side effects

Microsoft's indirect-injection guidance assumes some attacks get through and combines detection with isolation and least privilege. That is the correct starting point for an agent. The classifier can recommend a block; only the application can withhold a tool capability, reject an unauthorized resource, or require approval for a side effect. A final answer check cannot undo an operation that already ran. Microsoft: defense against indirect prompt injection.

Begin with the actual action boundary. Enumerate tools and resources, remove ambient credentials, and define what an attacker could change if the generator obeyed hostile text. Then decide where language-sensitive checks reduce that exposure. A read-only summarizer and a privileged code runner need different safeguards even when they use the same generator.

The practical requirement is an explicit safety decision process, not a universal obligation to run an extra neural model on every request. Start with deterministic boundaries and a measured baseline. Add specialized models when their supported task matches a remaining risk, and retain them only while evaluation shows useful error reduction at acceptable operational cost.

This is an engineering argument, not a claim that a particular law mandates a safety model. Legal duties depend on jurisdiction, system use, and the relevant obligations. A model purchase is not evidence that those obligations are satisfied. Even within engineering, document the decision not to add a classifier when the task is exact and the remaining exposure is contained. The useful artifact is a defensible control design with known limits, not a checklist asserting that every application must deploy the same stack.

The safety model engineering series

Existing detailed guides

Sources and evidence limits

Official sources checked on 29 October 2026. Capability descriptions reflect these documents, not independent model measurements. Check the pinned revision and current terms before deployment.

Views are my own and do not represent any employer, past or present.