Separate route quality from worker quality
A routing dashboard should show supported routes, escalations, unavailable specialists, worker errors and malformed results. Count these by service and taxonomy revision. A rise in unknown routes may mean customers discovered a new problem class, not that the router became worse. A rise in unavailable results usually points to deployment configuration. Combining both into a single assistant failure rate delays the correct repair.
Review a sample of successfully routed tickets as well as escalations. A confident wrong route can disappear from error dashboards while wasting specialist capacity. Label whether the selected specialty was appropriate, whether the handoff contained enough information and whether the returned evidence helped an operator. These are separate assessments. Keep ambiguous labels visible instead of forcing every incident into latency or availability merely because those are the first two categories implemented.
Architecture and request sequence
Read the architecture as text
- Route metrics: Quality and capacity by specialty
- Handoff audit: Scope and required fields
- Fault injection: Unavailable, error and invalid result
- Bounded fallback: Escalate without routing loops
- Taxonomy release gate: Compare labels and active tickets
- Route metrics → Handoff audit: sample route decisions
- Handoff audit → Fault injection: inject boundary failures
- Fault injection → Bounded fallback: apply fallback allowance
- Bounded fallback → Taxonomy release gate: compare taxonomy versions
Read the sequence as text
- Caller → Supervisor: Submit ambiguous symptom
- Supervisor → Router: Request route within taxonomy
- Router → Supervisor: Return unsupported route
- Supervisor → Review: Escalate with context
- Review → Caller: Explain supported manual path
Inject handoff and configuration failures
Remove a registered specialist while preserving the route allowlist, then confirm that the supervisor reports unavailable. Make a specialist raise an exception and return an invalid result in separate tests. Add a valid-looking locator whose object cannot be resolved when implementing a real evidence store. Each failure should remain distinguishable. A fallback that converts all of them to a generic explanation can make the system appear helpful while concealing a broken integration.
Test a handoff that omits the time window or service identifier required by the receiving specialist. The tiny fixture does not have a time window, so that test belongs to the expanded contract. Ensure the specialist refuses insufficient scope rather than selecting its own default service. Also test payloads from a previous schema revision during rolling deployment. Backward compatibility should be an explicit contract, not accidental tolerance of missing fields.
Treat the router as an untrusted proposer
Place an instruction in the symptom text asking for a privileged specialist or another service. The router might repeat that suggestion; the allowlist and service scope checks must still reject it. A production worker should not accept a raw route name as proof that its invocation was authorized. Bind the request to an authenticated caller, incident and allowed operation. This prevents a forged envelope from bypassing the supervisor simply by naming a valid specialist.
Review which context crosses the handoff. Sensitive customer details may be necessary for one specialty and irrelevant to another. Redact or replace them with authorized references when possible. The test that checks two envelope keys demonstrates minimization for the fixture only; it does not prove data-loss prevention in a model conversation. Add tests for the real serializers, transports and specialist tools that handle production content.
Manage asymmetric load and bounded fallback
A latency incident affecting many services can flood one specialist while others remain idle. Apply concurrency limits at the specialist boundary and expose queue age to the supervisor. When the budget is exhausted, return a delayed or escalated status with the collected context. Do not repeatedly reroute the same work to an unrelated specialist to improve apparent responsiveness. That trades a visible capacity problem for an invisible quality problem.
If a second specialist is genuinely useful, define when it is invoked and how the additional cost is charged. Preserve the first result and explain what question remains. Limit route hops and detect revisits. A pair of specialists that repeatedly recommend each other can consume unbounded calls even when each individual invocation has a short deadline. The parent incident needs a cumulative allowance and a stop condition independent of the workers' suggestions.
Deploy taxonomy changes as behavior changes
Adding a new specialist changes the router's decision space and may divert existing traffic. Evaluate the old and new taxonomy on the same labeled incidents, including cases near the new boundary. Monitor route distribution after release and compare operator overrides. Do not interpret a new route's high traffic as evidence of usefulness; it may reflect an overly broad description that attracts unrelated requests.
Before promotion, require that forbidden services never reach a specialist, unsupported routes remain escalations, handoffs preserve required scope and result failures retain their specific status. Test rollback with tickets already assigned to the new specialist. The old deployment may not know how to finish them. Keep a compatibility worker or an explicit migration and escalation path. Routing is part of the operational state, not merely a prompt that can be replaced without consequences.
Operational evidence and sources
The original supervisor implementation anchors the status vocabulary. The failure exercises and rollout policy are original proposals; Evals supplies evaluation prerequisites. No specialist service or live model was deployed. A production claim requires measured routing quality, tested transport authorization and incident-specific evidence validation beyond the fixture's string-prefix check.