Evidence label: deterministic synthetic model. The lab simulates a question-answering model rather than calling one. Each synthetic question has a difficulty that sets the true chance the model is right; the model sees a noisy version of that chance and reports a confidence that is sharpened (overconfident) to a degree you can change with the chat-tuned event. Multi-token answers add phrasing tokens whose probabilities have nothing to do with correctness, and the agreement signal samples k extra answers. The lab then computes ECE, AUROC, coverage and answered accuracy exactly as described in the article. All numbers are calculated from this synthetic model.
Play the lab
How to play
Choose a confidence signal, then adjust the calibration temperature T and bias b, which transform the log-odds as z / T + b. Set an answer threshold to abstain on low-confidence questions, and change the number of bins to see how the ECE estimate moves. The insight panel reports the temperature and bias that fit the current data best, which is what fitting on a labelled calibration set would give you. Inject a chat-tuned model, a new question domain or multi-token answers, and use the presets as reference points. Missions unlock in order; stars reward lower calibration error, higher coverage or a cheaper signal.
What the model assumes
Difficulty is normally distributed and maps to a probability of being right through a logistic curve, giving about 76% accuracy by default. The model's internal estimate is that probability's log-odds plus Gaussian noise, scaled by 1.35 (1.9 when chat-tuned) and shifted upward. In the new domain, the true probability falls by 1.6 log-odds units on half the questions, but the model's estimate does not. Multi-token answers have 1 to 12 tokens; extra tokens have probabilities between 0.55 and 0.99 unrelated to correctness. Agreement samples match the main answer with a probability driven by the model's internal estimate. Each model call costs an illustrative $0.002.
What it cannot tell you
Real miscalibration is not a single logistic distortion; it varies across topics, prompt formats and answer types, and semantic equivalence between differently worded answers matters for agreement signals. The lab ignores label noise, the cost of labelling, isotonic regression and conformal methods. Use it to understand what each metric and fix does, then measure calibration on your own labelled traffic.
Read the full explanation
- Logprobs and Calibration: Measuring When a Language Model Is Guessing (companion article)
- Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
- Constrained Decoding: How JSON Schema Mode Masks Invalid Tokens
All AI in Production labs · Lab catalog
- Guo et al., On Calibration of Modern Neural Networks (ICML 2017)
- Kadavath et al., Language Models (Mostly) Know What They Know
- OpenAI, GPT-4 Technical Report (calibration before and after post-training)
- Kuhn, Gal and Farquhar, Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (ICLR 2023)
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023)
- Geifman and El-Yaniv, Selective Classification for Deep Neural Networks (NeurIPS 2017)
Views are my own and do not represent any employer, past or present.