Put Safety Models at the Boundaries Where AI Agents Take Action

Safety model engineering / Guide 4 of 5. A source-linked decision guide with synthetic technical examples. No model benchmarks were run for this series.

Start with an operation map rather than a middleware checkbox

An agent can pass a final-answer moderation check after already performing an unauthorized operation. That is why safety belongs at decision boundaries, not only around the final string. Map the path from user input through retrieval, inference, proposed tools, tool results, and released output. For each edge, identify what data crosses a trust boundary and what action becomes possible.

Use a synthetic fixture agent with a read tool and a proposed write tool. The user has read permission for two named fixtures and no write permission. A retrieved file tells the agent to replace an output file. A content safety classifier might find no prohibited harm in that sentence. The application still rejects the operation because the capability was never granted. The example is a permission failure scenario, not a claim about any model's measured behavior.

NVIDIA's security guidance places authorization and parameter validation at the resource or API boundary. Microsoft's indirect-injection guidance similarly combines detection with least privilege and isolation. Those controls remain necessary even if an attack classifier returns benign. The architecture should tolerate at least some missed attacks instead of assuming perfect detection. NVIDIA: guardrails security guidelines; Microsoft: defense against indirect prompt injection.

Preserve source identity before asking a model to classify

Authenticate the principal and construct the permitted operation-resource set before inference. Give retrieval only the resources the principal can read. Enforce payload limits before expensive model calls, and retain the distinction between user instructions, retrieved documents, tool results, and generated proposals. A source label should come from application metadata, not from text that an attacker can edit.

Scan untrusted retrieved content against the appropriate attack detector, but keep it untrusted even after an allow verdict. A detector's job is to identify suspicious content within its documented scope. It does not promote a document into an instruction source. Benign security documentation can contain attack examples, and a hostile instruction can be phrased in ordinary technical language.

Short-context detectors require an explicit segmentation policy. Prompt Guard 2 has a 512-token window. Overlapping windows are a useful design candidate, but maximum-window score is only an aggregation heuristic. Test attacks spanning the boundary, long benign quotations, and non-English content. Do not silently truncate the only part of the document that contains the attack. Meta: Prompt Guard 2 86M model card.

Authorize the parsed proposal immediately before execution

# Synthetic in-memory fixture tools. No filesystem access occurs.
FIXTURES = {"/fixtures/a.txt": "alpha", "/fixtures/b.txt": "beta"}
GRANTS = {("read", "/fixtures/a.txt"), ("read", "/fixtures/b.txt")}

def execute(proposal):
    if not isinstance(proposal, dict):
        raise ValueError("proposal must be an object")
    if set(proposal) != {"operation", "resource"}:
        raise ValueError("unexpected argument contract")
    operation, resource = proposal["operation"], proposal["resource"]
    if not isinstance(operation, str) or not isinstance(resource, str):
        raise ValueError("arguments must be strings")
    if (operation, resource) not in GRANTS:
        raise PermissionError("operation not granted")
    return FIXTURES[resource]

assert execute({"operation": "read", "resource": "/fixtures/a.txt"}) == "alpha"
try:
    execute({"operation": "write", "resource": "/fixtures/a.txt"})
except PermissionError:
    pass
else:
    raise AssertionError("write must be rejected")

The fixture intentionally has exact resource identities and no filesystem path resolution. A real filesystem adapter must canonicalize paths safely, constrain roots, address symlinks and race conditions, and re-check permissions against the operation actually executed. Do not copy this small demonstration as a complete filesystem sandbox.

The model's safety verdict is absent from the grant set on purpose. A model can recommend rejecting a proposal, but it cannot add permission. Validate strict schemas, supported operation names, argument types, bounds, and resource ownership. Use narrowly scoped credentials at execution time rather than placing broad credentials in the model's context or environment.

Bind approval to the exact normalized proposal, not just to the conversation. If an operation, resource, or significant argument changes after approval, require the new proposal to satisfy policy again. Re-check expiring grants immediately before the side effect, and avoid a check-then-use gap in which a resource changes between validation and execution. For external operations, use resource-native authorization, transaction boundaries, and idempotency keys where supported. These controls address timing and identity; a language classifier cannot infer them reliably from a friendly-looking description of the proposed action.

Tool results need a second untrusted-data boundary

A tool can return syntactically valid data that contains hostile instructions, unsupported facts, secrets, or irrelevant material. Preserve the tool's identity and resource provenance, validate expected formats and sizes, and apply relevant content or attack checks before feeding the result into another inference step. A successful tool call does not make its text authoritative instructions.

Framework integration details matter. NVIDIA's current AgentMiddleware documentation states that its checks inspect content only: tool-call arguments are not inspected, and ToolMessage inputs bypass input rails. That documented limitation does not imply the framework is useless; it means the application must cover those boundaries explicitly. Read the hooks and known limitations for the exact version in use rather than equating middleware installed with complete coverage. NVIDIA: Agent Middleware and known limitations.

LlamaFirewall illustrates another distinction: it combines scanners with different roles, including prompt attack detection, alignment checks, and static code analysis. The framework is not a single model whose score covers all threats. Configure and test each relevant control, and avoid treating a scanner result as proof that a generated command or returned code is safe to execute. Meta: LlamaFirewall architecture.

Define failure behavior before the guard becomes unavailable

The example actions are design choices, not a universal enforcement policy.
Outcome	Meaning	Example application action
Supported allow	The selected policy check passed	Continue within existing permissions
Block	The check detected a policy violation	Stop that path and provide an appropriate explanation
Review	A defined uncertainty band was reached	Queue a bounded review if a review path exists
Unsupported	The model cannot assess this input or category	Route to a supported checker or refuse that feature
Error	Timeout, malformed output, or dependency failure	Bounded retry or explicit safe degraded mode

A safety timeout should not silently become unrestricted generation. Conversely, availability requirements may justify a limited read-only mode with clearly bounded output instead of a total shutdown. Choose behavior by the consequence of the operation, document it, and test it under dependency failures. A configured review state is meaningful only if someone or something can actually complete the review.

Retries need budgets. Repeating the same unsafe request through a different model until one allows it is not a robust fallback. Keep the policy constant, cap attempts, retain the reason for escalation, and prevent a late result from reviving a cancelled operation. Idempotency and cancellation remain deterministic application concerns.

Buffer output before a verdict can arrive too late

Streaming changes the exposure boundary. A token-level detector can signal while generation is underway, but it cannot retract bytes already sent to a user or tool. Define the unit buffered before release, the classifier's required context, and what happens when a block arrives after partial output. Larger buffers can reduce exposure while increasing perceived latency. Measure both.

Qwen3Guard distinguishes completed-input Gen models from incremental Stream models. That distinction makes a candidate available for different serving designs; it does not eliminate the application's buffering and cancellation responsibilities. An output filter also cannot compensate for an unauthorized tool invocation earlier in the loop. Qwen: Qwen3Guard repository.

Record minimized operational events: boundary name, policy version, checkpoint revision, threshold, outcome, reason code, timing, and correlation identifier. Avoid retaining raw prompts by default. Shadow new guard versions before enforcing them, inspect disagreements on benign and adversarial slices, and keep a rollback path. The result is a layered execution system in which model checks inform decisions while exact controls contain side effects.

The safety model engineering series

Existing detailed guides

guardrails architecture
llm sec prompt injection scanners

Sources and evidence limits

Official sources checked on 29 October 2026. Capability descriptions reflect these documents, not independent model measurements. Check the pinned revision and current terms before deployment.

Views are my own and do not represent any employer, past or present.