Reasoning Models and Thinking Budgets: When Extra Tokens Buy Accuracy

What a thinking budget actually controls

A reasoning model generates a hidden or separately delimited chain of reasoning before its visible answer. Training methods such as the reinforcement learning described in the DeepSeek-R1 report reward correct final answers, and longer, self-checking reasoning traces emerge because they help. At inference time, you usually have one of three controls:

• A token ceiling on thinking. The model reasons until it decides to stop or until the ceiling, at which point it must move to the answer.
• A coarse effort level. Low, medium or high, which the provider maps to how much the model is encouraged to think. The model still decides per request.
• Budget forcing. Directly controlling the end of thinking: inserting the end-of-thinking delimiter to stop early, or, as in the s1 paper by Muennighoff and colleagues, appending a word like "Wait" when the model tries to stop so it keeps reasoning.

The important distinction is between a ceiling and a target. A ceiling only binds when the model would have thought longer; a target makes every request spend roughly the budget. Most production settings should be ceilings, with forcing reserved for measured cases.

The arithmetic: tokens, dollars and seconds

Thinking tokens are output tokens. They cost output prices and they are generated one at a time at decode speed. With illustrative numbers of $10 per million output tokens and 150 tokens per second:

reasoning step  ~ 240 tokens (lab assumption)
easy problem    : 2-4 steps   ->   500-1,000 thinking tokens
medium problem  : 5-9 steps   -> 1,200-2,200 thinking tokens
hard problem    : 11-20 steps -> 2,600-4,800 thinking tokens

cost per request  = (thinking + answer) x $10 / 1M + input x $2 / 1M
4,000 thinking tokens -> $0.040 per request -> $40 per 1,000 requests
latency           = (thinking + answer) / 150 tok/s
4,000 thinking tokens -> ~28 s before the last answer token

Two consequences follow. First, average cost is driven by the traffic mix: if a quarter of requests are hard, they account for a disproportionate share of thinking tokens, because each one needs several times the reasoning of an easy request. Second, tail latency is set by the hardest requests: p95 latency is roughly the 95th percentile of thinking length divided by decode speed, which is why a budget that looks cheap on average can still break a latency SLO.

Why accuracy rises and then flattens

OpenAI's o1 announcement showed accuracy on hard benchmarks rising roughly linearly with the logarithm of test-time compute. The shape is intuitive once you think in reasoning steps. Below the number of steps a problem needs, the model is forced to answer from an incomplete argument and does little better than guessing; at the threshold, accuracy jumps; above it, extra tokens go into re-checking, which catches some slips but adds little. Averaged over a mix of difficulties, these jumps smooth into the familiar log-shaped curve.

There is a second effect at the top of the curve. Studies of o1-style models, such as Chen and colleagues' work on overthinking, found that models often spend many tokens on easy problems, re-solving and re-verifying answers they already had, with little or no gain. Re-checking is not free of risk either: each extra pass is another chance to talk itself out of a correct answer. In the lab, raising the budget from 8k to 16k tokens with a ceiling policy raises cost by about 60% and lowers accuracy slightly, because the model uses spare budget for more verification and occasionally flips right answers. The chart in the lab plots accuracy against budget for the current policy so you can see where your setting sits on that curve.

All values are calculated by the lab's synthetic model with 25% hard traffic; they illustrate the shape, not any real model.

Budget forcing: when to make the model think longer

The s1 paper showed that a small model fine-tuned on a thousand reasoning examples could be pushed to better scores on competition maths by suppressing its end-of-thinking token and appending "Wait", and that forcing an early stop reduced scores, giving a controllable trade-off. That is a useful research result and a risky production default. Forcing every request to spend the full budget multiplies cost on easy traffic and stretches latency, and on problems the model had already solved it mostly adds opportunities to change a right answer.

In the lab, the force policy at 16k tokens costs about $165 per thousand requests, against about $41 for an 8k ceiling, with lower accuracy, because easy problems are re-litigated until the budget runs out. Mission three loads exactly this setup and asks you to keep accuracy at 95% or more for under $70 per thousand. Forcing earns its place for narrow, high-value tasks where you have measured a gain, such as a final verification pass on a small set of critical answers.

Routing by difficulty

If hard requests need large budgets and easy ones do not, the obvious move is to decide the budget per request. Snell and colleagues studied compute-optimal test-time scaling and found that the best strategy depends strongly on prompt difficulty, and that allocating compute adaptively can be several times more efficient than a uniform strategy. In production, routing can be as simple as a cheap classifier, rules on input features (length, presence of code or numbers, the product surface) or a first pass at low effort that escalates when the model signals uncertainty.

Routing has its own failure mode: misclassification. A hard request routed to a small budget is cut short, and the error is invisible unless you measure accuracy by route. The lab's router sends 10% of hard problems and half of medium ones to a quarter of the budget, which is why its accuracy can trail a plain ceiling: the ceiling already lets easy problems stop early, so routing mainly saves the re-checking tokens that a generous ceiling invites. Measure both on your traffic before assuming routing wins.

Parallel samples and majority voting

Self-consistency, introduced by Wang and colleagues, samples several reasoning paths and takes the most common final answer. It works because independent errors rarely agree. If one sample is right with probability 0.9 and wrong answers spread over several values, three samples give the right majority with probability of about 0.97 from the two-or-three-correct cases alone, plus a little from three-way ties broken at random.

The cost is linear in the number of samples, but the latency is not: samples run in parallel, so p95 latency follows the slowest sample rather than the sum. That makes voting a latency tool as much as an accuracy tool. In the lab, a single trace with a 16k budget plateaus near 95% at a p95 of about 56 seconds, while three samples at 5k reach about 98.6% at about 34 seconds, for more money. Mission four asks you to find that combination. Voting needs answers that can be compared, such as numbers, labels or normalised short strings; for free-form text you need a judge or a similarity measure instead.

The knobs in real systems

Controls differ by provider and change often, so treat this as orientation and check current documentation:

• Effort levels. OpenAI's reasoning models accept a reasoning_effort setting such as low, medium or high. Current Claude models use adaptive thinking, where the model decides how much to think, steered by an effort setting in the request's output configuration; earlier Claude models took an explicit budget_tokens ceiling with a minimum of 1,024 that had to be below max_tokens.
• Token ceilings. Some APIs expose a thinking-token budget directly; check whether it is a hard cap or a guide, and whether max_tokens includes thinking tokens, because a low max_tokens can cut off the answer after a long think.
• Open-weight serving. Many open reasoning models delimit thinking with special tags. vLLM and SGLang provide reasoning parsers that separate reasoning from the answer in responses; some model chat templates offer a switch to disable thinking entirely. Budget forcing is implemented by stopping generation at a token count and appending the end-of-thinking delimiter, which you can do with your own logic.
• Billing and visibility. Thinking tokens are typically billed as output even when they are hidden or summarised. Log them separately from answer tokens.

A worked example: Belgavi's Book Store support assistant

Suppose the hypothetical Belgavi's Book Store runs a support assistant on a reasoning model. Most questions are easy (order status, return policy), a quarter involve multi-step reasoning about bundled discounts and loyalty points, and the team has a p95 latency target of 45 seconds for the reasoning route. With a 1,024-token ceiling, the lab's model answers most easy questions correctly but cuts short almost every medium and hard one: about 43% overall accuracy. Raising the ceiling to 4,096 lifts accuracy to about 88% at roughly $28 per thousand requests.

Then a promotion launches and 60% of questions become hard. At 4,096 tokens, a fifth of attempts are cut off and accuracy falls to about 76%. Raising the ceiling to about 6,000 tokens brings accuracy back above 90% within the 45-second p95 target, at around $41 per thousand. The team should not jump to 32k "to be safe": the curve is flat there, latency is higher, and cost per thousand roughly doubles. The durable fix is monitoring: alert on the share of requests whose thinking hit the ceiling, because that number moves before accuracy complaints arrive. Mission two replays this surge.

Measure cost per correct answer

Budget decisions go wrong when teams compare cost per request. A cheaper request that is wrong is not cheap: it triggers a retry, a human escalation or a lost customer. The right unit is cost per correct answer, ideally with the cost of a wrong answer added explicitly. If a wrong answer at Belgavi's Book Store costs a support agent five minutes, even a few points of accuracy can justify a much larger thinking budget; if wrong answers are cheap to detect and retry, a smaller budget plus a retry on low confidence can win.

A practical evaluation loop looks like this: sample a few hundred real requests with known answers, bucket them by difficulty using whatever signal you have, run each budget or effort level you are considering, and record accuracy, thinking tokens and latency per bucket. Plot accuracy against tokens for each bucket. The knee of each curve is the budget that bucket needs; the gap between buckets tells you whether routing is worth building. Re-run the sweep whenever the model version changes, because both the token cost per step and the shape of the curve move between versions.

Failure modes and how to detect them

• Starved reasoning. Many requests end exactly at the thinking ceiling. Track the share of responses whose thinking length equals the budget; a rising share means the traffic has become harder.
• Answer truncation. A long think consumes max_tokens and the visible answer is cut. Alert on length-terminated responses.
• Overthinking spend. Thinking tokens per request rise after a model or effort change while accuracy is flat. Compare tokens by difficulty bucket, not only on average.
• Latency tails. p95 and p99 latency track the longest thinking traces; set timeouts and user-facing progress accordingly.
• Silent model updates. A new model version can spend more tokens per step for the same work, which shifts every budget. The lab's wordier model update event models a 40% increase.

Common mistakes include evaluating budgets on a benchmark whose difficulty mix differs from production, comparing models at different effort settings, and treating the provider's default effort as tuned for your workload.

Run the lab: Budget the thinking: buy accuracy without paying for overthinking

The lab runs 360 synthetic problems of three difficulties through a model that reasons in steps of about 240 tokens. The top of the canvas shows six sample traces against the budget (dashed line), coloured by difficulty; the chart below plots single-sample accuracy against budget for the current policy. Start with mission one, where a 1,024-token budget leaves hard problems guessing.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Log thinking tokens and answer tokens separately for every request, with the effort or budget that was set.
2. Measure the share of requests whose thinking reached the budget, by route, and alert when it rises.
3. Run an accuracy-versus-budget sweep on a sample of real traffic, bucketed by difficulty, before choosing a default.
4. Check that max_tokens leaves room for the answer after the longest expected thinking.
5. Try a lower effort or ceiling on easy routes and compare cost per correct answer, not cost per request.
6. For questions with comparable answers, test three parallel samples with majority voting against a single longer budget at equal latency.

Sources and further reading

• s1: Simple test-time scaling (Muennighoff et al., 2025)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al., 2024)
• Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al.)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Do NOT Think That Much for 2+3=? On the Overthinking of o1-like LLMs (Chen et al., 2024)
• OpenAI: Learning to reason with LLMs

Related in this series:

• Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
• Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
• Logprobs and Calibration: Measuring When a Language Model Is Guessing

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.