Why architecture matters here
The economics are brutal and simple. Frontier-model tokens cost one to two orders of magnitude more than small-model tokens, and latency scales with model size. If 70% of traffic is easy — a typical figure for assistants with a broad user base — then routing that share to a model a tenth the price cuts the bill by more than half before any other optimization. No amount of prompt-caching or batching cleverness recovers the money left on the table by sending 'what's a good pasta recipe' to your most expensive deployment.
But the reason routing deserves real architecture, rather than an if-statement on prompt length, is the asymmetry of errors. Overrouting (easy query to the big model) wastes money linearly. Underrouting (hard query to the small model) produces a wrong answer, a frustrated user, or a silent hallucination — costs that are nonlinear and reputational. A router is therefore not maximizing average cost efficiency; it is managing a quality floor under a cost objective. That framing drives every design choice: calibrated confidence rather than argmax classification, escalation paths rather than one-shot dispatch, and per-segment quality SLOs rather than a single global win rate.
There is also an organizational forcing function: model fleets churn. New checkpoints ship monthly, providers deprecate endpoints, prices change quarterly. Hard-coding model choice into application code means every change is a code change across every feature. A router centralizes the fleet decision behind one policy surface — applications declare what they need (task type, quality tier, latency budget), and the router decides which model satisfies it this month. That indirection is what lets platform teams swap models under stable product behavior.
Latency budgets push the same direction. Interactive surfaces live and die on time-to-first-token; a router that sends chit-chat to a small co-located model turns a 900-millisecond experience into a 150-millisecond one, which user studies consistently show matters more than marginal answer quality for simple turns. Batch surfaces — overnight document processing, eval runs — invert the trade: latency is nearly free, so the router can run cascades aggressively and buy quality with retries. One routing layer serving both regimes needs the latency budget as a first-class input, not an afterthought.
The architecture: every piece explained
The feature extractor turns a raw request into routing signals: task type (classification, extraction, creative, code, math), prompt and context length, conversation depth, presence of tool schemas, user tier, and — most predictive in practice — an embedding of the request compared against clusters of historically easy and hard traffic. Cheap and fast is the constraint: the extractor runs on every request, so it must cost milliseconds, not model calls.
The router policy consumes those signals. Three implementations dominate, in ascending order of sophistication. Heuristic rules — task-type and length thresholds — are transparent and a fine v1. Embedding classifiers — nearest-neighbor or a logistic head over the request embedding, trained on pairs of (request, which models answered it acceptably) — capture semantic difficulty and retrain cheaply. Learned router models — a small transformer fine-tuned to predict per-model win probability — squeeze out the last few points of accuracy at the cost of a real training pipeline. The output in all cases should be calibrated probabilities, not a hard label, because the downstream decision weighs confidence against cost and the escalation budget.
The policy engine wraps the classifier with business constraints: per-tenant budgets, latency SLOs (a request with a 2-second budget cannot ride the big model regardless of difficulty), compliance pinning (this customer's data only touches this deployment), canary percentages for new models, and kill switches. Keeping policy separate from prediction is what lets operators change behavior at runtime without retraining anything.
A route decision is more than a model name: it includes generation parameters (temperature, max tokens, reasoning-effort settings on models that expose them), the prompt-template variant for that model family, and the fallback chain if the primary deployment is unhealthy. Treating the full tuple as the routing output — and logging it as one versioned object — is what makes routed traffic reproducible when someone asks why last Tuesday's answers looked different.
Downstream sit the model tiers and the quality gate. Cascade architectures skip prediction entirely: every request hits the small model first, and a verifier — a self-consistency check, a judge model scoring the answer, or task-specific validation like unit tests for code — decides whether to accept or escalate. Cascades never underroute silently (everything gets checked) but pay double latency on escalated traffic. Predictive routing avoids the double hop but eats misclassification risk. Mature systems blend both: predict the tier, verify cheap-tier outputs, escalate on failure — with the escalation controller enforcing a ceiling (at most one retry, budget permitting) so a pathological request cannot climb the ladder unboundedly. Every decision, score, and escalation lands in the routing log, which is simultaneously the audit trail, the eval set generator, and the router's training data.
End-to-end flow
Trace one request. A user asks a product assistant to 'compare these two contract clauses and flag conflicts with our standard terms.' The gateway authenticates, loads tenant policy (enterprise tier: quality-sensitive, generous budget, 8-second latency SLO), and hands the request to the feature extractor: task type scores as reasoning-heavy analysis, context length is 6k tokens, the embedding lands near a cluster of historically hard legal-comparison traffic. Extraction takes 15 milliseconds.
The router model emits calibrated win probabilities: small model 0.42, mid model 0.78, frontier 0.94. The policy engine applies constraints — no compliance pinning, latency budget admits any tier, tenant budget is healthy — and picks the mid model: it clears the 0.75 quality-floor threshold at a third of frontier cost. The decision, with all features and probabilities, is stamped into the request context for logging. Dispatch adds the model-specific prompt template and calls the mid-tier deployment.
The response streams back through the quality gate. For this task type the gate runs two cheap checks in parallel: a refusal/deflection detector (did the model actually answer?) and a judge-model spot check that scores answer completeness against the request on a 1–5 scale. The judge returns 2 — the answer compared the clauses but missed the conflict-flagging half. The escalation controller consults its ladder: one escalation permitted, budget check passes, latency spent so far leaves room. It re-dispatches to the frontier model, this time with the failed draft included as context ('a previous attempt missed the conflict analysis'), which both improves the retry and costs less than a cold start because much of the context is cache-warm at the provider.
The frontier answer scores 5, streams to the user, and the full trace — features, route, scores, escalation, per-hop cost and latency — lands in the routing log. Overnight, that trace joins thousands of others: escalated requests become hard-labeled training pairs ('mid model insufficient here'), sampled non-escalated traffic gets judge-scored for calibration drift, and the weekly retrain nudges the boundary between mid and frontier for legal-comparison traffic. The user saw one answer, 9 seconds, no visible seam.
Two flow details separate production routers from prototypes. First, session stickiness: mid-conversation model switches change tone and capability visibly, so the router pins a conversation to its current tier unless difficulty shifts decisively — re-evaluating per turn but requiring a margin (hysteresis) before switching, exactly like a thermostat avoiding rapid cycling. Second, cache interplay: provider-side prompt caching makes the previously-used model cheaper than list price for follow-up turns in the same conversation, which the policy engine must price in — the naive per-token comparison overestimates the savings of switching down and causes routers to thrash between tiers that caching had already equalized.