Why architecture matters here
The architecture matters because the two naive alternatives are both structurally wrong, and in opposite ways. Route everything to the strong model and you have built a system whose cost scales with total traffic at the highest per-request price, most of it spent on requests the cheap model would have answered identically. In a workload where 80% of requests are easy, you are overpaying by roughly the price gap on four-fifths of your volume — a tax you pay forever, growing with adoption. Route everything to the cheap model and you have capped your quality at the cheap model's ceiling; the hard tail — the requests where a good answer is worth the most — gets the worst treatment, and users learn the agent is unreliable on exactly the tasks they care about.
The router resolves this because request difficulty is both skewed and cheaply estimable. The skew means there is a large pool of easy requests whose cost you can slash without touching quality. The estimability means you can identify them before committing: a short question with no code, a lookup with an obvious answer, a request matching a well-covered intent — these are recognizable by a small classifier far cheaper than the model it gates. The router monetizes the gap between 'how much it costs to decide where to send this' and 'how much it costs to send it everywhere,' and that gap is enormous when the deciding is cheap and the sending is expensive.
Routing also matters because capability is not one-dimensional. A request is not just 'easy or hard' — it may need a specific skill. A SQL question is best served by an agent with database tools and a schema in context; a math problem by one with a code interpreter; a legal question by one grounded in the right corpus. A single general model handles all of these worse than a specialist would, and no amount of prompting turns a generalist into a database agent with live schema access. The router is how a system composed of specialists presents a single front door: it reads the request and dispatches to the specialist whose tools and grounding fit, so the user gets expert handling without having to know which expert to ask.
Finally, the router converts an implicit, unmanaged tradeoff into an explicit, tunable one. Without a router, the cost-quality balance is whatever your one model happens to give — you cannot dial it. With a router, the balance is a set of thresholds you can move: lower the escalation bar and quality rises as cost rises; raise it and you save money at some quality risk. Every route becomes a measurable segment with its own cost and quality, so you can see exactly what each dollar of routing decision buys and adjust deliberately rather than hoping one model's defaults happen to fit.
The architecture: every piece explained
Top row: from request to decision. An incoming request — the user's turn plus relevant context — first hits the classifier, whose job is to extract the features that determine routing: what is the intent (question, code task, data query, chit-chat), how hard does it look (length, presence of code, ambiguity, required reasoning depth), and does it need a particular skill? The classifier can be a small model, a fast heuristic, an embedding-nearest-neighbor against labeled examples, or a blend; its defining constraint is that it must be much cheaper than the handlers it selects, or the routing overhead eats the savings. Its output feeds the routing policy — the rules and/or learned model that map features to a handler — which produces both a chosen route and, critically, a confidence in that choice. The confidence gate is where the router decides whether it trusts its own decision enough to act, or whether the request is ambiguous enough to escalate to a stronger handler or a human rather than guess.
Middle row: the handlers. The routing policy dispatches to one of several destinations. The cheap model takes the easy bulk — the requests the classifier is confident are simple — at low cost and low latency. The strong model takes the hard requests where the extra capability is worth the price. A specialist agent takes domain requests that need particular tools or grounding — a code agent with an interpreter, a SQL agent with schema and a database connection, a math agent. And human handoff takes the requests that are too risky, too ambiguous, or too low-confidence to hand any model — a safety valve for the tail the automation should not attempt. The set of handlers is the palette; the router's skill is matching each request to the right one.
The confidence gate deserves emphasis because it is what separates a robust router from a brittle one. A router that always commits to its top choice will confidently misroute the ambiguous requests — precisely the ones it understands least. A router with a confidence threshold instead escalates on uncertainty: when the classifier's signal is weak, it defaults up the capability ladder (to the strong model or a human) rather than down. This asymmetry is deliberate. Misrouting an easy request to the strong model costs a few cents; misrouting a hard request to the cheap model costs a wrong answer to someone who needed a right one. The gate is tuned to make the cheaper error, not the balanced one.
Bottom rows: the feedback loop and operations. A router is only as good as its policy, and the policy decays as traffic drifts, so the selected handler runs, produces an answer, and the outcome — was it correct, did the user retry, did it need escalation after all — flows into a feedback loop that retrains or retunes the policy. Routing is a learning problem: yesterday's misroutes are today's training labels. The ops strip names the disciplines that keep it honest: routing accuracy measured against outcomes, cost and quality tracked per route so you can see what each destination delivers, a fallback path when a chosen handler fails or the router mis-decides, and drift monitoring that catches the request distribution shifting out from under a policy trained on last month's traffic.
End-to-end flow
Trace four requests through a router with a cheap model, a strong model, a SQL specialist, and a human handoff.
Request 1 — easy, routed cheap: 'What does HTTP 502 mean?' The classifier sees a short factual question, no code, a well-covered intent, and high confidence that it is easy. The policy routes to the cheap model, which answers correctly in 400ms for a fraction of a cent. This is the router doing its main job — serving the easy majority at low cost — and it happens on the vast bulk of traffic invisibly.
Request 2 — hard, routed strong: 'Given these three conflicting requirements, design a caching strategy and justify the tradeoffs.' The classifier flags multi-step reasoning, ambiguity, and no single right answer — high estimated difficulty. The policy routes to the strong model, which spends more tokens and more time to produce a genuinely reasoned answer. The system paid the premium exactly where it buys quality, and only here.
Request 3 — domain, routed to a specialist: 'How many orders shipped late last quarter by region?' The classifier detects a data query needing schema knowledge and a live database. The policy routes not to any general model but to the SQL specialist agent, which has the schema in context and a database connection, generates and runs a query, and returns real numbers — something no general model could do without those tools. The router's value here is not cost but capability matching: it sent the request to the only handler that could actually answer it.
Request 4 — low confidence, escalated to a human: 'Cancel my enterprise contract and refund the last three invoices.' The classifier recognizes a high-risk action request, and the policy's confidence in automating it is deliberately low — the cost of an automated mistake here is severe. The confidence gate escalates to human handoff rather than guessing. The human resolves it; the outcome is logged as 'correctly escalated,' reinforcing the policy.
The loop closes: each outcome flows back. Request 1's quick success confirms the easy route; Request 2's quality confirms the escalation was worth it; Request 3's valid query confirms the specialist match; Request 4's clean human resolution confirms the risk gate. Over time these labeled outcomes retrain the policy, and drift monitoring watches for the day the request mix shifts — more code tasks, say — so the policy is retuned before its accuracy quietly erodes. The router is not a fixed map; it is a system that gets more accurate the more it routes.