Separate refused work from failed work
An overloaded gateway can protect the service by refusing new work. That refusal should not be counted as a successful incident analysis, nor should it be confused with a model failure after dispatch. Track admission decisions, dispatched calls, completed calls, canceled calls, and uncertain outcomes separately. Report the denominator for every success or cost metric so cheaper averages do not conceal a growing population of rejected requests.
The assistant's user-facing response should explain whether the requested analysis was performed. A fallback that returns a generic incident checklist has changed the task. Label that change and retain the original request state. Otherwise operators can mistake a degraded answer for completed diagnosis, making the low-cost route appear more successful than it actually is.
Architecture and request sequence
Read the architecture as text
- Admission telemetry: Separate refusal from dispatched failure
- Usage reconciliation: Measure charges and aged reservations
- Route quality evaluation: Compare eligible tasks and coverage
- Reviewed routing policy: Bound changes and retain rollback path
- Admission telemetry → Usage reconciliation: Classified signal
- Usage reconciliation → Route quality evaluation: Reviewed evidence
- Route quality evaluation → Reviewed routing policy: Controlled response
Read the sequence as text
- Runtime → Monitor: Report scoped observations
- Monitor → Review: Check coverage and failures
- Review → Operator: Submit evidence for review
- Operator → Runtime: Authorize bounded next step
- Runtime → Monitor: Record observed outcome
Inject pressure without contacting a model
The local test suite can exhaust the single active slot and budget deterministically. That is useful for validating decision order without spending provider quota. For a real integration, run a bounded load exercise that measures actual queue delay, active calls, cancellation latency, and reservation errors. Use synthetic incident data and approved credentials; a local loop of dictionary calls cannot establish cloud throughput.
Test a request whose deadline expires while queued, a provider response lost after processing, and a worker restart with outstanding reservations. Each scenario needs a stated ledger result. Unknown remote usage must not automatically refund the entire reservation. If reconciliation is unavailable, pausing further work can be more accurate than presenting a balance that the service cannot justify.
Make retry policy part of admission
Immediate retries from many callers can sustain the overload they are trying to escape. Return a bounded retry signal when the condition is temporary, and have clients respect it within their remaining deadline. The retrying operation should preserve its identity so the admission service can distinguish a new task from another delivery of the same one. Do not let the model improvise backoff timing from a natural-language error.
The 429 specification provides a standard rate-limit response, but the application still owns retry policy and budget semantics. A depleted monthly entitlement differs from a busy worker pool. Map those cases deliberately, and test that the client stops when its allowed attempts or deadline are exhausted.
Measure quality and cost for each eligible route
A routing change can reduce spending while increasing unsupported incident conclusions. Evaluate each route on the task classes it is allowed to handle, including counterexamples and difficult cases. Track quality conditional on route alongside the admission rate and coverage of the evaluation set. A route that processes only easy cases should not be compared directly with one assigned every ambiguous incident.
Version the routing table, eligibility rubric, model revision, and cost estimator. Keep a baseline and a candidate policy for controlled replay over the same fixtures. A deterministic replay of stored provider responses evaluates orchestration decisions; it does not prove that a live model will produce identical content. Record that distinction when presenting a policy improvement to the release owner.
Observe budget drift and isolate privilege
Expose reserved units, settled units, outstanding reservation age, overage discrepancies, and reconciliation backlog. Alert when old reservations accumulate even if remaining budget looks healthy. A leaked reservation can deny legitimate work; an unjustified refund can permit overspend. The service needs an operator procedure for both conditions, with an audit trail for manual adjustments.
Only an authorized accounting or policy service should change limits. A prompt that says 'this incident is important, ignore the budget' cannot raise a tenant's entitlement. Keep provider credentials outside model-generated tool arguments, restrict permitted destinations, and log the resolved route rather than secrets. A budget boundary limits cost exposure; it does not replace authorization over what the agent is allowed to do.
Accept a policy only with a recovery path
Before enabling a new router, demonstrate that capability floors hold under budget pressure, concurrency never exceeds the intended admission bound, duplicate requests do not double-reserve, and uncertain charges remain visible. Then test restart recovery and operator reconciliation against the actual storage implementation. A successful local test run is the first layer of that evidence, not the final deployment sign-off.
For a proposed rollout, start with shadow routing decisions, compare them with the current policy, and canary only a bounded eligible population after review. Retain a way to disable the candidate without losing active reservation records. The desired outcome is a predictable service under scarcity: it performs an authorized task within a known contract or refuses it with an accurate state.