Design model routing with capability floors and bounded admission

Pattern 13 companion: Python example · Behavioral tests · Example license

Cheap work is not always admissible work

During a major outage, an incident-response assistant receives both short status-summary requests and complex requests to compare conflicting diagnostic evidence. Sending everything to the most expensive model can exhaust capacity. Sending everything to the cheapest model can violate the task's requirements. This original design separates eligibility from optimization: first identify routes allowed for the task, then choose an affordable eligible route.

The model names in the example are synthetic fast and deep labels. Their capability levels and unit reservations are teaching values, not measurements of real models. A real routing table needs evaluations tied to specific model revisions, task classes, tool requirements, and data policies. Cost cannot establish eligibility by itself, and a confident classifier cannot authorize a lower capability floor than the application permits.

Architecture and request sequence

Component diagram for Design model routing with capability floors and bounded admission — Original reference architecture by Sandeep Belgavi. Arrows show the responsibilities and transitions described below.

Request sequence for Design model routing with capability floors and bounded admission — Illustrative sequence for this design; failure and recovery variants are explained in the article.

Reserve capacity before dispatching the call

A budget check followed by a separate model invocation can race with another request. Both callers can observe the same remaining balance and spend it. The proposed production boundary reserves expected work atomically with admission, then settles against measured usage. The local exercise implements the arithmetic sequentially in one process. It does not claim a distributed transaction or thread-safe admission service.

Our illustrative router begins with seven units, reserves five for a deep request, and refuses another request while one slot is active. When the first request settles at four units, one unused unit returns. These units are abstract and cannot be interpreted as tokens, seconds, or currency without an explicit pricing model. Recording the unit definition prevents a plausible dashboard from becoming misleading financial accounting.

Set a capability floor that survives overload

A high-requirement incident analysis must not silently fall back to a route whose documented acceptance set excludes that task. If the only eligible route is unaffordable, return a budget denial or a controlled escalation. A lower-cost answer can still be offered as a clearly different operation, such as summarizing already approved evidence, if the caller accepts that contract.

This distinction also applies to privacy and tool support. A route that meets quality goals but cannot process a tenant's data is ineligible. A route that lacks a required structured-output feature needs a different adapter or should be excluded. Keep eligibility reasons inspectable. The model adapter receives a resolved route after the gate, rather than selecting a provider and asking the budget system for forgiveness afterward.

Bound queueing and reject expired work

A request that cannot finish usefully before its deadline should not wait indefinitely while holding scarce resources. The exercise checks whether the supplied deadline budget is already nonpositive at admission; it does not predict runtime or enforce a wall-clock timeout. A production scheduler needs actual deadline propagation and cancellation semantics across model and tool calls.

Google SRE describes graceful degradation and load shedding under overload and warns that request counts can poorly represent resource demand. Source: handling overload. For this proposed incident service, track admitted work units and active calls as well as request count. A short summary and a long multi-tool investigation should not consume identical scheduling assumptions merely because both arrived as one HTTP request.

Return a refusal that callers can handle

The caller needs to distinguish expired, overloaded, and budget-denied decisions. Retrying an expired operation without a new deadline cannot help. Retrying overload immediately can amplify pressure. A budget denial might require a smaller authorized task or an approved budget increase. The status must be machine-readable so the client does not infer retry behavior from a model-generated explanation.

RFC 6585 defines HTTP 429 for excessive request rates and permits a Retry-After field. Source: HTTP rate-limit status. That does not imply every budget refusal should use the same HTTP status. A reference API must map its own admission categories deliberately and document whether the client can retry, wait, or request a different operation.

Capacity assumptions belong in the contract

Assume a single active call in the teaching router, two units for fast, and five for deep. The design demonstrates bounded accounting, not an optimal routing algorithm. A real system needs empirical latency distributions, provider limits, quality acceptance sets, and reservation error measurements. Underestimated reservations must remain visible; do not hide overspend by reducing an unrelated tenant's balance.

For multi-tenant operation, combine tenant budgets with a global capacity limit and define priority entitlements outside the prompt. Protect emergency work without allowing every request to claim emergency status. The acceptance question is whether every dispatched call had an eligible route and a valid reservation, including during overload. That property is more useful than a low average cost that conceals denied or degraded work.