Why architecture matters here
Router architecture matters because model cost varies by 20-50x across tiers. Sending every query to the top model is expensive; sending every query to the cheap model degrades quality. The router is the balance point.
Cost impact is direct and large.
Reliability comes from confidence + escalation. When the router isn't sure, escalate; when the cheap model isn't confident, escalate. Users experience the top model's quality on hard queries only.
The architecture: every piece explained
Walk the diagram top to bottom.
User Request. Arbitrary prompt with context.
Router. A classifier or rules engine that decides which model tier handles this.
Model Pool. Cheap (small model, cents per query), mid (mid-tier), top (frontier model).
Intent classifier. Small model or heuristic that predicts query difficulty. Trained on labeled examples of "easy" vs "hard."
Cost + latency budget. Per-tier caps. If budget tight, favor cheaper.
Small model handles. Simple lookups, factual, standard chat.
Mid model handles. Moderate reasoning, code assistance, summarization.
Top model handles. Multi-step reasoning, high-stakes decisions, complex code.
Escalation on low confidence. If small model's confidence low, resend to mid or top.
A/B routing evaluation. Compare user satisfaction across tiers to refine router.
End-to-end routing flow
Trace a request. User: "What's the capital of France?"
Router classifier scores as "simple factual" with 95% confidence. Route to small model.
Small model answers "Paris." Cost: $0.0001.
Second request: "Design a distributed caching system for a global app with strong consistency."
Classifier scores as "complex reasoning" with 90% confidence. Route to top model.
Top model produces detailed response. Cost: $0.01.
Third request: "Fix this bug in my code" with 500 lines. Classifier uncertain (65%). Route to mid; if response's self-reported confidence low, escalate to top.
Metrics: over a week, 70% simple → small model, 25% moderate → mid, 5% hard → top. Weighted average cost per query: $0.0005 vs $0.01 if all top. 20x savings.
A/B: user satisfaction ratings tracked. Small-model satisfaction on simple queries: 4.6/5. Mid: 4.7. Top: 4.7 on hard. No degradation.