Evidence label: deterministic synthetic model. This page runs a small JavaScript model of a reasoning model under a thinking budget. Each of 360 synthetic problems needs a number of reasoning steps that depends on its difficulty; each step costs a random number of tokens and can introduce a slip. The model stops when it has finished and re-checked, re-checks more when it has spare budget, and must answer early if the budget runs out. Extra re-checking can fix slips or occasionally flip a right answer. Several samples can vote. Nothing here runs a real model; all numbers are calculated estimates.
Play the lab
How to play
Choose a policy (ceiling, forced full budget, or difficulty routing), a thinking budget, the number of parallel samples and the share of hard problems. Every change re-runs the same seeded problems, so differences come from your change. Watch accuracy overall and on hard problems, thinking tokens per request, cost per thousand requests, p95 latency, the share of attempts cut off by the budget and the share that over-thought. Events raise hard traffic to 60% or make every reasoning step 40% wordier.
Missions unlock in order; each loads a broken setup. Stars reward the cheaper answer once the goal is met.
What the model assumes
Easy problems need 2-4 steps, medium 5-9 and hard 11-20; each step costs 150-330 tokens. Each needed step introduces a slip with probability 0.8% (1.2% for hard). After finishing, the model re-checks for 0-2 steps plus 30% of any spare budget; each re-check fixes an outstanding slip with probability 0.3, or flips a correct answer with probability 1% (3.5% when forced). An attempt cut short answers correctly with probability 0.25 × (fraction done)². Wrong answers take one of three values for voting. Prices are an illustrative $10 per million output and $2 per million input tokens; decode speed is 150 tokens per second.
What it cannot tell you
Real models do not reason in tidy fixed-size steps, difficulty is not labelled, and the relationship between budget and accuracy differs by model and task. Provider effort levels are not simple token ceilings. Use the lab to understand the shape of the trade-off and the failure signatures, then sweep budgets on your own traffic.
Read the full explanation
- Reasoning Models and Thinking Budgets: When Extra Tokens Buy Accuracy (companion article)
- Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
- Model Cascades and Routing: Cutting LLM Cost Without Losing Answer Quality
All AI in Production labs · Lab catalog
- s1: Simple test-time scaling (Muennighoff et al., 2025)
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al., 2024)
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al.)
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Do NOT Think That Much for 2+3=? On the Overthinking of o1-like LLMs (Chen et al., 2024)
- OpenAI: Learning to reason with LLMs
Views are my own and do not represent any employer, past or present.