Choosing Reasoning Effort per Request: A Routing Policy and an Evaluation Harness

One global setting is the wrong unit

Reasoning models expose a control for how much hidden reasoning to spend before answering. Many applications set it once, usually to the default, and never revisit it. That leaves money and latency on the table in one direction and quality in the other: a classification request and a multi-step debugging request get the same budget.

The useful unit is the task class. A routing policy maps each class of request to a starting effort, and a verifier decides whether to escalate. The policy is chosen from evaluation data, not intuition, and it is re-checked when the model changes. This article describes the controls as documented on 17 October 2026, a harness for choosing levels, and the escalation pattern. For routing across different models rather than effort levels, see agent request router architecture; for how reasoning tokens are priced, see LLM pricing models.

What the controls look like today

Three documented details shape the design. First, supported values and defaults are model-specific. OpenAI's guide lists models that reject none with HTTP 400 and models that default to medium; Gemini's table lists different default levels per model. A router must therefore hold a capability table per model and validate the level before sending, or it will fail at run time after a model upgrade.

Second, reasoning tokens are billed as output tokens and occupy context, even though they are not returned. Third, OpenAI's max_output_tokens caps reasoning plus visible output together. If the cap is hit, the response comes back with status incomplete and reason max_output_tokens, possibly before any visible text, so you pay for reasoning and get nothing usable. OpenAI recommends reserving at least 25,000 tokens for reasoning and output while experimenting. These are the kinds of facts that change between releases, so link your capability table to the provider pages and re-verify when you change models.

The policy: classes, starting levels and an escalation ladder

Start with a small number of task classes that you can recognise cheaply, from the endpoint, the tool set in play, or a lightweight classifier. For each class, record a starting effort, a maximum effort, a latency budget and a verifier. The verifier is a deterministic check that can judge the output without another model where possible: JSON schema validity, unit tests passing, a SQL query that parses and runs against a sample, a citation that resolves.

At run time, call at the starting level. If the verifier fails, escalate one level and retry, up to the maximum and within the latency budget. This spends high effort only on the requests that need it, and the escalation rate per class becomes a monitoring signal: when it climbs, the inputs or the model changed.

The table is an illustrative starting point, not a recommendation backed by measurements. Your levels come from the harness below.

The evaluation harness

For each task class, assemble an evaluation set from real, labelled traffic. Run every item at every candidate level several times, because outputs vary between runs. Record the verifier outcome, wall-clock latency, reasoning tokens and total cost. Then choose the cheapest level whose pass rate is good enough with statistical confidence, using a lower confidence bound instead of the raw mean, so that a lucky run on a small set does not pick a level that fails in production.

Illustrative python, checked for syntax; adapt names and storage to your system.

import math
from dataclasses import dataclass

@dataclass
class LevelResult:
    level: str
    passes: int
    trials: int
    p95_latency_s: float
    mean_cost: float

def wilson_lower(passes: int, trials: int, z: float = 1.96) -> float:
    """Lower bound of the Wilson score interval for a pass rate."""
    if trials == 0:
        return 0.0
    phat = passes / trials
    denom = 1 + z * z / trials
    centre = phat + z * z / (2 * trials)
    margin = z * math.sqrt(phat * (1 - phat) / trials + z * z / (4 * trials * trials))
    return (centre - margin) / denom

def choose_level(results, target: float, latency_budget_s: float):
    """Cheapest level whose pass-rate lower bound meets the target within the latency budget."""
    ok = [r for r in results
          if wilson_lower(r.passes, r.trials) >= target and r.p95_latency_s <= latency_budget_s]
    return min(ok, key=lambda r: r.mean_cost).level if ok else None

# Illustrative numbers only: 120 items x 3 runs per level.
runs = [LevelResult("low", 318, 360, 3.1, 0.0021),
        LevelResult("medium", 344, 360, 6.8, 0.0049),
        LevelResult("high", 349, 360, 14.0, 0.0112)]
print(choose_level(runs, target=0.92, latency_budget_s=10.0))

With these illustrative numbers, low passes 88.3 per cent with a lower bound near 0.85, so it misses a 0.92 target; medium passes 95.6 per cent with a lower bound near 0.93 and meets the latency budget; high is better still but too slow for this class. The function returns medium. Repeat the exercise for the escalation path: measure the pass rate of "low, then medium on verifier failure" as a policy, because that is what you will actually run, and it is often cheaper than a fixed medium setting while matching its quality.

Budgets that do not truncate silently

Give every call an explicit output cap derived from the class, not a single global value. Size it from observed reasoning-token percentiles at that level plus the longest acceptable visible answer, with headroom. Treat an incomplete response with reason max_output_tokens as a distinct outcome: log it, count it, and either escalate the cap once or fail the request with a clear message. Never pass a truncated answer to the user as if it were complete.

Track cost per successful outcome, not cost per call. An escalation policy makes some requests cost two calls; that is fine if it raises the success rate or lowers the average cost compared with always running at the higher level.

Failure modes

• Unsupported level after a model upgrade. A router that sends none to a model that rejects it fails every request in that class. Validate against the capability table and test the table in CI.
• Empty incomplete responses. A low output cap at high effort produces paid responses with no visible text.
• Evaluation sets dominated by easy items. Every level passes and the cheapest wins, until hard production traffic arrives. Stratify the set by difficulty and report per stratum.
• Comparing levels across model versions. A level chosen for one model snapshot says nothing about the next. Re-run the harness when the model changes.
• Verifiers that are too lenient. Escalation only works if failures are detected. A schema check that passes on empty arrays lets bad extractions through at low effort.
• Latency tails. Higher effort widens the latency distribution. Use p95 or p99 in the selection rule, not the mean.

Rolling it out

1. Log effort level, reasoning tokens, latency and verifier outcome for every call before changing anything.
2. Define task classes and verifiers, and build the capability table per model from the provider documentation.
3. Run the harness per class and pick starting and maximum levels.
4. Ship the policy behind a flag for a slice of traffic and compare cost per successful outcome and user-visible failure rate with the old global setting.
5. Alert on escalation rate and incomplete-response rate per class; both move first when something changes.

When a policy is not worth it

If your traffic is one homogeneous task, a single level chosen with the harness is enough. If volume is low, the engineering time costs more than the tokens. And if no cheap verifier exists for a class, escalation has nothing to trigger on; pick one level for that class from offline evaluation and spend effort on building a verifier instead.

Practice and related reading

Further reading:

• Agent request router architecture
• LLM pricing models
• AI evaluation frameworks

Sources and scope

Checked against these sources on 17 October 2026. Examples and figures are illustrative, not measurements.

• OpenAI API documentation: reasoning models and reasoning effort
• Gemini API documentation: thinking
• Wilson (1927): Probable inference, the law of succession, and statistical inference