Evidence label: deterministic synthetic model. The lab is a JavaScript model of compute-optimal sizing. It converts GPU-hours to FLOPs, evaluates a Chinchilla-style parametric loss surface (with refitted coefficients and a repeated-data discount), simulates noisy pilot training runs at three small budgets, fits a parabola to each and extrapolates the optimum, and adds a simple serving-cost estimate. No model is trained and no GPU is measured; every number is a calculated estimate.
Play the lab
How to play
Choose a training budget and drag the model size along the curve. The tiles show predicted loss, the share of the budget effectively wasted compared with an optimally sized model, tokens per parameter, total training plus serving cost, how far the pilot fit is from the true optimum, and how much compute the pilots consumed. Add lifetime serving tokens to see how inference cost changes the best choice, and change the number of pilot runs to see how the extrapolation behaves. Use a new random seed to see how much the pilot fit depends on luck.
Missions unlock in order and reward the most efficient answer. The events add a data wall of 0.15T unique tokens, noisier pilot runs, and a cluster that runs at 60% of planned speed for the whole job.
What the model assumes
Loss follows L = E + A/N^alpha + B/D^beta with E = 1.817, A = 482, B = 2085, alpha = 0.348 and beta = 0.366, the refit published in a 2024 replication of Chinchilla's parametric approach. Compute is 6ND. Repeated data is discounted with an exponential decay over about 15 repetitions, following the data-constrained scaling study. Pilot runs add Gaussian noise to the predicted loss. Serving cost assumes 2N FLOPs per token at 30% utilisation of an H100's dense BF16 peak and $2.50 per GPU-hour.
What it cannot tell you
The coefficients come from one family of models, one tokenizer and one data mix; yours will differ, and the absolute losses mean nothing outside the lab. There is no learning-rate tuning, no architecture choice, no continued pretraining from a checkpoint, and no downstream evaluation, which is what you actually care about. Use the lab to build intuition for the shape of the trade-offs and run IsoFLOP pilots on your own data before committing a budget.
Read the full explanation
- Compute-Optimal Model Sizing: Applying Chinchilla Scaling Laws to a Real Budget (companion article)
- Mixture-of-Experts Routing: Load Balancing, Capacity Factors and Dropped Tokens
- LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work
All AI in Production labs · Lab catalog
- Training Compute-Optimal Large Language Models (Chinchilla)
- Scaling Laws for Neural Language Models
- Chinchilla Scaling: A replication attempt
- Scaling Data-Constrained Language Models
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
Views are my own and do not represent any employer, past or present.