Guardrail Outages and Fail-closed Design: Keep Useful Service Without Unsafe Actions

Unavailable is a decision state

A policy service times out during a refund request. If the application treats the exception as an empty result and continues, the outage becomes an authorization bypass. If it blocks the entire product indiscriminately, users may lose access to harmless read-only help. The correct behavior depends on the capability and the evidence required to use it.

Represent unavailable separately from allow and deny. Deny means the policy evaluated the request and rejected it. Unavailable means the system cannot establish the required decision. The user experience and operational response should reflect that difference without exposing sensitive internals.

Create a capability-specific fallback table

Define this table before an incident. Otherwise developers may add emergency bypasses under pressure without considering the protected effect. Keep fallback behavior in reviewed configuration and test it alongside normal policy.

Restricted service should actually remove risky capabilities. A banner saying “safety checks unavailable” does not make an unrestricted tool safe. If the fallback is read-only, use read-only credentials and a tool inventory that cannot perform writes.

Bound retries and deadlines

Set a total request deadline and allocate portions to screening, authorization, generation, and execution. Propagate cancellation so a timed-out parent stops launching new child work. Avoid independent retry loops at every layer, which can multiply calls during an outage.

Use jitter and backoff for retryable transport failures, but do not retry permanent denials as if they were transient. A malformed policy response should trigger an integration error and containment, not a random retry with fewer fields. Track retries under one logical operation identifier.

Implement a deliberate fallback decision

Illustrative python; adapt the persistence and identity adapters to your service.

def fallback(capability, missing_check):
    if capability in {"refund", "change_identity", "send_message"}:
        return "hold_action"
    if capability == "public_faq" and missing_check == "model_screening":
        return "approved_static_answer"
    return "unavailable"

assert fallback("refund", "authorization") == "hold_action"
assert fallback("public_faq", "model_screening") == "approved_static_answer"

This example is intentionally a product decision table rather than a universal fail-open or fail-closed switch. The static answer must be previously approved and independent of untrusted personalized context. Do not generate a new answer and label it static merely because it came from a cache.

When a held action resumes, revalidate all time-sensitive evidence. Do not execute a queued payment automatically after a long outage using an expired approval or old balance. Recovery changes the operating context and requires current checks.

Avoid bypasses through caches and fallbacks

A cache can improve availability, but a cached authorization decision has an expiry and invalidation problem. Cache immutable public content freely according to its lifecycle; treat personalized data and allow decisions more carefully. Include principal, tenant, resource, policy version, and relevant evidence version in the cache contract.

A fallback model is another execution path. It must preserve required tool restrictions and output handling. If the primary model fails, do not switch to a helper that has broader permissions or skips screening. Test fallback behavior with the same abuse fixtures as the primary path.

Exercise partial failures

Inject timeouts before and after a service accepts a request. Test a classifier returning an unsupported result, an authorization service returning malformed JSON, a quota reservation succeeding while the response is lost, and a worker crashing during cleanup. Assert the final state and the absence of unintended effects.

Run recovery tests as well as outage tests. Restore the dependency and verify that held operations are reconciled, expired approvals remain expired, and clients do not flood the system with accumulated retries. A circuit breaker closing is not proof that the backlog can safely execute.

Worked incident: authorization service fails during queued refunds

A policy dependency becomes unavailable while several approved refunds wait in a queue. The approvals were valid when created, but execution requires a current eligibility check. The worker should hold the operations and report dependency_unavailable, not assume that a previous approval is sufficient to bypass the missing check.

Keep reservations and operation state consistent with the workflow's contract. If no external submission occurred, the system may safely retain or release a reservation according to policy. If submission may have occurred, use uncertain state and reconciliation. Do not apply the same cleanup rule to both cases merely because both surfaced as a timeout.

During the outage, the assistant can show authoritative pending status and approved static guidance. It should not tell users to resubmit repeatedly or create fresh operation identifiers. Client retries should refer to the existing operation. This reduces both duplicate risk and recovery load.

When the dependency recovers, process held operations with current checks and controlled concurrency. Expired approvals require renewed review; changed balances may make some proposals ineligible. The recovery worker should not drain the queue under stale assumptions just to restore a green latency dashboard.

Test this sequence in staging by pausing the policy adapter, advancing synthetic time beyond some approval expiries, and then restoring it. Assert that fresh eligible operations can complete, expired ones remain held, and no duplicate processor submissions occur. The drill demonstrates safe degradation and safe recovery, which are separate properties. A system that blocks during an outage but executes stale actions afterward is still vulnerable.

Monitor the cost of safety-related unavailability

Track unavailable decisions by dependency and capability, user-visible delay, held-action age, and recovery backlog. Alert on unexpected execution during a mandatory-check outage. That metric can reveal a bypass even when the service's availability dashboard looks healthy.

The objective is predictable degradation: preserve useful low-risk service while preventing actions whose prerequisites cannot be verified. A well-designed outage policy is part of AFP because attackers and ordinary failures can both exploit ambiguous error handling. Make the safe behavior testable before the dependency fails in production.

Practice and continue

Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.

• Guardrail Metrics: Precision, Recall, Escape Rate, and Review Burden
• AFP Observability and Incident Response: Reconstruct Decisions Without Leaking Data

Sources and implementation scope

Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.

• ADK safety and security
• OWASP authorization