Agentic systems / Pattern 10 / Handle failure and scale

Operate agent gateways with explicit denial and delegation evidence

Detect authority drift, preserve useful audit trails and avoid turning outages into privilege escalation.

By Sandeep Belgavi · 23 October 2026 · Architecture and design patterns

Denial is a normal system outcome

A tool gateway will reject requests during ordinary operation: an investigation may outlive a credential, a target may move to another tenant, or the assistant may ask for a tool outside its assigned role. Treat these as typed outcomes rather than generic provider failures. Otherwise an automatic retry loop may repeatedly submit the same unauthorized request or search for a broader tool that bypasses the intended boundary.

Define the caller's response to each denial. Expired credentials may require a trusted refresh flow; a missing scope may require a separately authorized workflow; a wrong tenant should stop the request. The model can explain the limitation to the user, but it should not manufacture a new identity or claim that another credential is equivalent. Authority changes need a controlled path outside conversational improvisation.

Architecture and request sequence

Component diagram for Operate agent gateways with explicit denial and delegation evidence
Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.
Read the architecture as text
  • Identity health: Expiry and verification failures
  • Policy inventory: Effective tools and resources
  • Gateway telemetry: Allows, denials and budgets
  • Provider circuit: Bounded remote failures
  • Audit review: Subject, actor and decision
  • Identity health → Gateway telemetry: trusted or rejected
  • Policy inventory → Gateway telemetry: current policy
  • Gateway telemetry → Provider circuit: allowed requests
  • Gateway telemetry → Audit review: decision evidence
  • Provider circuit → Audit review: provider outcome
Request sequence for Operate agent gateways with explicit denial and delegation evidence
Illustrative sequence for this design; failure and recovery variants are explained in the article.
Read the sequence as text
  1. Gateway → Operator: Report denial-rate change
  2. Operator → Gateway: Inspect safe reason categories
  3. Operator → Audit: Compare actor and policy version
  4. Gateway → Provider: Use bounded authorized read
  5. Operator → Gateway: Pause affected access path

Make delegation visible in audit

An audit entry should preserve both the subject on whose behalf work occurs and the agent actor performing it. Include the operation, resolved resource identity, tenant, policy version, decision, reason and correlation identifier. Avoid storing raw access tokens or complete sensitive tool payloads. The reference records only a minimal in-memory subset to make its tests transparent.

Durable auditing is a deployment requirement, not an automatic property of appending to a Python list. Decide what happens if audit storage is unavailable for a consequential operation. A read-only service may have a different policy from a remediation service, but the choice must be explicit. Test that log enrichment cannot accidentally include secrets from exception messages, request headers or provider responses.

Watch for authority drift

A previously narrow role can grow as teams add tools and exceptions. Review the effective operations and resource scope, not only the role's name. A role called incident-reader may eventually include a generic execution tool that defeats its original purpose. Compare policy versions and run the denial suite whenever a tool schema, resource resolver or identity mapping changes.

The companion tests make several boundaries executable: wrong audience, wrong actor, cross-tenant target and write requests must fail before provider access. Extend them with real resource ownership fixtures and revoked identities when implementing authentication. Also test a valid read credential presented to a different gateway. A successful signature check should not cause the receiving service to ignore audience or resource restrictions.

Handle outages without broadening privilege

If a provider health endpoint fails, the assistant may propose using a shell, database console or administrative API to obtain the same information. That substitute can carry far more authority. Keep fallback selection inside an explicit policy: a transient read failure should not automatically grant write access or unrestricted execution. If no authorized fallback exists, return an unavailable result with an appropriate escalation path.

Set bounded retries and circuit behavior at the adapter. Track denial rates separately from provider timeouts so an authorization problem is not mistaken for infrastructure instability. MCP security guidance also addresses scope minimization and confused-deputy risks. Use that guidance for protocol-specific implementation work while retaining application-level tests for which incident resources the assistant may actually reach.

Control throughput and credential exposure

Measure allowed calls, denied calls by safe reason category, provider latency, concurrent requests and repeated requests per incident. Count cost after authorization as well as model inference before it. An agent can create a large bill through many legitimate reads even when every request is individually permitted. Apply per-tenant and per-incident budgets that cannot be reset by changing a model-generated correlation identifier.

Keep credential acquisition in trusted infrastructure and use credentials scoped to the downstream service. Do not expose tokens in the model context as a convenience for troubleshooting. A provider adapter should return safe diagnostic categories rather than raw authentication errors that may contain sensitive configuration. Redaction is a backstop, not a substitute for preventing secrets from entering the conversational path in the first place.

Acceptance and rollout evidence

Start with the read-only synthetic adapter, then connect a test environment with a small resource allowlist. Verify the entire chain: invalid authentication is rejected before policy, denied authorization is rejected before provider access, allowed reads produce an auditable result and no credential appears in traces. Rehearse expiry, identity-provider outage, policy-store outage and resource ownership changes before granting wider access.

Use the evaluation layer to measure whether the assistant chooses useful operations and interprets results correctly, while keeping the gateway's denial tests independent. A passing reasoning benchmark cannot compensate for a cross-tenant access bug. The release decision should state the actual authority granted, tested failure modes and remaining boundaries. The goal is a system whose mistakes remain constrained and explainable, not a claim that the model will never request the wrong tool.

Sources and implementation