A training run streams one number at you more than any other: the loss. Watching it fall — or fail to — is the single richest, cheapest diagnostic you have, and a practitioner who can read the shape of a loss curve can catch a doomed run in the first thousand steps instead of the last. The trouble is that ‘the loss is going down’ is not a diagnosis. A healthy curve, a learning rate set ten times too high, a rate ten times too low, a corrupt batch, and quiet overfitting all produce distinct, recognizable fingerprints, and each points to a different fix. This piece is a field guide to those shapes. We start from what the number actually means — cross-entropy, perplexity, and the ln(vocab) anchor that tells you a run started correctly — then walk the failure modes one by one, pair the loss with the gradient norm, and close with a set of concrete numeric cues you can check against your own logs. The goal is to turn a wall of scrolling numbers into a decision.
The loss is your primary instrument
Almost everything you can know about a run mid-flight is encoded in two scalar streams: the training loss and the validation loss. Weights, activations, and attention maps are richer, but they are expensive to inspect and hard to read; the loss is one float per step, logged for free, and its trajectory over time carries a startling amount of signal. The skill worth building is reading the shape rather than the instantaneous value.
Two habits make the shape legible. First, plot on a log-scale y-axis (or plot ln(loss)): healthy language-model training decays roughly as a power law, and a power law is a straight line on a log-log plot, so deviations from ‘straight’ jump out. Second, smooth the curve — an exponential moving average over ~50–100 steps — because the raw per-step loss is noisy and the noise itself is information you want to separate from the trend. Once you can see the trend and the noise band as two distinct things, most of the diagnoses in this article become a matter of pattern-matching against the characteristic shapes below.
What the number means: cross-entropy, perplexity, bits
Language-model loss is the cross-entropy between the model’s predicted next-token distribution and the one-hot truth: for a token with predicted probability p, the loss contribution is -ln(p), averaged over all positions. Three equivalent readings of that number are worth memorizing because they turn an abstract scalar into intuition.
loss = -(1/N) Σ_i ln p(x_i | context) # nats/token
perplexity = exp(loss) # effective branching factor
bits_per_tok = loss / ln(2) = loss × 1.4427 # nats -> bits
loss 2.0 -> ppl ≈ 7.4 bits ≈ 2.89
loss 3.0 -> ppl ≈ 20.1 bits ≈ 4.33
loss 1.0 -> ppl ≈ 2.72 bits ≈ 1.44Perplexity is the model’s effective ‘how many choices am I guessing between’ — exp(loss). Bits-per-token is the same information in compression units: a loss of 2.0 means the model needs ~2.89 bits to encode each token. These conversions matter for diagnosis because they make a loss value checkable against reality — you know a coherent English model lands somewhere around 1 bit per character, and a value that implies nonsense (perplexity in the thousands late in training) is a red flag, not just ‘a bit high.’
The single most useful cue: ln(vocab) at step zero
Before you diagnose the shape, check the starting height. At initialization the model has learned nothing, so it should assign roughly uniform probability 1/V to all V vocabulary tokens. The cross-entropy of a uniform distribution over V outcomes is exactly ln(V), so a correctly initialized run must start its loss near ln(vocab_size).
expected step-0 loss ≈ ln(V)
V = 50257 (GPT-2 BPE) -> ln(V) ≈ 10.82
V = 32000 (Llama SP) -> ln(V) ≈ 10.37
V = 256 (byte-level) -> ln(V) ≈ 5.55This one number catches a whole class of bugs before you waste a single GPU-hour. A run that starts far below ln(V) — say a byte-level model opening at 2.0 instead of ~5.55 — is almost always leaking the label into the input (an off-by-one in the shift, or a mask that lets the model peek at the token it is predicting). A run that starts far above ln(V) usually has a broken initialization (weights too large, logits exploding). If step zero is wrong, nothing downstream is trustworthy, so make this the first thing you assert in any new training pipeline.
The healthy curve: steep drop, then power-law decay
A well-configured run has an unmistakable profile. In the first 100–1000 steps the loss falls steeply as the model learns the cheap structure — unigram frequencies, the fact that spaces and common words dominate, basic positional habits. From a ln(V) start of ~10.8 it might reach 4–5 quickly. Then the descent bends into a long, smooth, gradually flattening decline: the power-law regime where each further halving of loss costs exponentially more compute.
Three properties mark it as healthy. The trend is monotone under smoothing — minor per-step wiggles are fine, but the moving average should not trend up. The gradient norm is stable, hovering in a bounded band rather than swinging by orders of magnitude. And the validation loss tracks training loss with a small, roughly constant gap. On a log-scale plot the mature phase looks nearly linear. When people say a run ‘looks good,’ this straight-on-log, small-val-gap, stable-gradient shape is what they mean — and every failure mode below is a specific, named departure from it.
A gallery of shapes
Before dissecting each pathology, it helps to see them on one axis. The sketch below overlays the four you will meet most often: the healthy power-law decay, the too-slow crawl of a learning rate set well below its useful range, the too-hot run that dips briefly and then diverges to NaN, and the quietly treacherous overfit, where training loss keeps sliding down while validation loss bottoms out and turns back up.
Keep this mental picture handy. Diagnosis in practice is mostly overlaying your actual curve on this gallery and asking which fingerprint it matches: is it flattening too high (blue), climbing after a dip (pink), or splitting from the validation line (amber)? Each shape below gets its own section with the numeric cues and the specific knob to turn.
Learning rate too high: divergence, spikes, NaN
The learning rate is the parameter the loss curve is most sensitive to, and the too-high signature is dramatic. Mild overshoot shows up as a curve that descends but is jagged and unstable, riddled with upward spikes it only partly recovers from. Push further and the curve diverges: it drops for a few hundred steps, then bends upward and climbs, often terminating in inf or NaN as activations overflow float range. The pink curve in the gallery is this exact story.
The mechanism is straightforward: each step takes a gradient stride proportional to the rate, and when strides are too large the optimizer overshoots the local basin and lands somewhere worse, compounding step over step. The tell that distinguishes it from other spikes is the gradient norm exploding in lock-step with the loss — norms jumping to 10×, 100× their baseline just before the blow-up. Fixes, in order of first resort: cut the peak LR (halving is a reasonable first step), lengthen warmup so the model reaches the high rate more gently, add or tighten gradient clipping (a max-norm around 1.0 is standard), and verify your numerical precision — pure fp16 without loss-scaling is a classic NaN factory that bf16 largely cures.
Learning rate too low: slow, flat, underpowered
The opposite error is quieter and easy to tolerate for too long. With the rate set well below its useful range the curve is smooth and stable but descends far too slowly — the blue line in the gallery, still high and only gently sloping when it should be deep into the power-law regime. There are no spikes and nothing looks broken, which is exactly why it wastes budget: the run is healthy, just crawling.
The cue is comparative and quantitative. Ask where the loss should be by a given step, given your model size and token budget, and notice that you are sitting a full unit of loss above it with a nearly flat slope. A too-low rate also leaves the gradient norm small and placid — the optimizer is barely moving. The fix is to raise the peak LR (often you can go several times higher than a timid first guess) and, ideally, to have found the right neighborhood ahead of time with a short LR range test: ramp the rate exponentially over a few hundred steps and watch where the loss falls fastest just before it starts to diverge. That fastest-descent band, backed off slightly, is your target — the border between the blue curve and the pink one.
Loss spikes: read them before you react
An isolated spike — the loss jumps sharply for one or a few steps — is common even in otherwise healthy large-model runs, and the right response depends entirely on what happens next. A spike that recovers within tens of steps back to the pre-spike trend is usually benign: a bad batch (a corrupt document, a pathological run of repeated tokens, an extreme outlier) briefly threw a large gradient, clipping absorbed most of it, and the run healed. Note it and move on.
A spike that does not recover — the loss settles onto a permanently higher plateau, or begins a slow climb — means the optimizer was kicked out of its basin and the run is damaged. This is when you restart from the last good checkpoint. The pragmatic recipe used on large runs: keep frequent checkpoints, and on an unrecovered spike, roll back to the checkpoint before it, then either skip the offending data shard or resume with a briefly lowered LR to slip past the rough patch. Do not restart on every wiggle — over-reacting to benign spikes wastes as much time as ignoring real ones. The discriminator is recovery: watch a few dozen steps before you touch anything.
Skip-step and clipping: automated spike defense
Because spikes are routine at scale, mature training loops defend against them automatically rather than relying on a human watching the dashboard. Two mechanisms do most of the work, and both key off the gradient norm rather than the loss, because the norm moves first.
Gradient clipping rescales the whole gradient vector whenever its global norm exceeds a threshold (commonly 1.0): if the norm is 50 and the cap is 1.0, every component is scaled by 1/50, preserving direction while bounding step size. This turns what would have been a divergent overshoot into a merely large-but-survivable step. Skip-step logic goes further: if the gradient norm exceeds a much larger bar — say several multiples of its running mean, or any non-finite value — the loop discards that step entirely, applying no update, and continues. The batch that produced a norm of 10,000 simply never moves the weights. Together these make the loss curve robust: on the same data that would blow up a naive loop, a clipped-and-skip-guarded run shows a brief blip and carries on. If you see frequent skips in the logs, though, treat it as a symptom — the underlying LR or data still needs attention.
Overfitting: train down, validation up
Overfitting is the failure the training loss alone cannot show you, which is why validation loss earns its keep. The signature is a divergence between the two curves: training loss keeps falling while validation loss reaches a minimum and then rises — the amber curve in the gallery, splitting upward from a training line that continues down. The model has stopped learning generalizable structure and started memorizing the training set, so it looks ever-better on data it has seen and steadily worse on data it has not.
The correct action is early stopping: the best model is the one at the validation minimum, not the final step, so keep the checkpoint at that trough (marked in the sketch). If overfitting arrives too early to reach acceptable quality, the levers are more data, stronger regularization (weight decay, dropout), a smaller model, or fewer epochs. One crucial caveat keeps this honest: classic train-down/val-up overfitting is largely a fine-tuning and small-data, multi-epoch phenomenon. In single-epoch large-scale pretraining, where the token budget vastly exceeds parameter count and the model sees each example roughly once, you typically do not see the validation curve turn up — train and validation stay glued together. Seeing the split is itself a signal that you are re-using data harder than the model can absorb.
Underfitting: high and flat on both curves
Underfitting is the mirror image and is easy to confuse with a too-low learning rate, so the distinction matters. Here both training and validation loss are high and have flattened, with a small gap between them — the model is not memorizing anything, it simply lacks the capacity or the training to represent the data well. The curve settled onto a plateau that is plainly too high given what comparable models achieve.
Distinguish the causes by the gap and the gradient. If train and val are high, close together, and the gradient norm has gone quiet, the model has genuinely converged to a poor optimum: the fix is more capacity (a bigger model), longer training, or a better architecture — not more regularization, which would only push loss higher. If instead the curve is high but the gradient norm is still healthy and the slope is merely gentle, you are likely LR-limited (the previous section’s blue curve) and should raise the rate before concluding the model is too small. The trap is adding regularization to fight a high loss that is actually underfitting — that makes it worse. High-with-small-gap says ‘do more,’ not ‘constrain more.’
Plateaus: when to wait and when to worry
A plateau — the loss flattening for a stretch and then resuming its descent — is one of the most misread shapes, and the usual instinct to kill the run is often wrong. Training frequently proceeds in tiers: the model masters one class of structure, sits at a temporary floor while it reorganizes, then unlocks the next tier and the loss drops again. Killing a run during such a plateau throws away the breakthrough that was about to happen.
The discriminator is, again, the gradient norm. If the loss is flat but the gradient norm is non-zero and active, the model is still moving through parameter space — the optimization is working, and patience is warranted. If the loss is flat and the gradient norm has collapsed toward zero, learning has genuinely stalled and waiting will not help; look at the LR schedule (has it decayed to near-nothing too early?), the data, or the model size. A related benign plateau appears at the very end of a cosine or linear decay schedule, where the LR approaches zero by design and the loss naturally flattens as the steps shrink — that flattening is the schedule finishing, not the model failing.
Warmup: the fingerprint of the first few hundred steps
The opening of the curve is shaped as much by the learning-rate warmup as by the model. Warmup ramps the rate linearly from near zero up to the peak over the first 500–2000 steps (often a few percent of total training) because a fresh model with a poorly conditioned loss landscape cannot tolerate full-size steps immediately — hit it with the peak rate at step one and you get the pink divergence.
On the curve, healthy warmup looks like a slightly gentler initial descent that steepens as the rate reaches its peak, with the gradient norm rising in a controlled way rather than spiking. Two failure fingerprints are worth recognizing. Too-short warmup produces an early spike or wobble right around the point where the rate hits its peak — the model was rushed. Too-long warmup wastes budget: the loss descends sluggishly for thousands of steps because the effective rate is still tiny, mimicking a too-low-LR crawl until the ramp finally completes. When a run diverges specifically in its first few hundred steps, lengthening warmup is frequently a more surgical fix than cutting the peak rate, because it preserves the steady-state learning speed while smoothing only the fragile opening.
Batch size: how it reshapes the noise band
Batch size changes the curve in two visible ways, and understanding both keeps you from misreading them. First, noise: each step’s gradient is estimated from a finite sample, and the estimate’s variance falls as the batch grows. Larger batches therefore produce a smoother, thinner-banded loss curve; small batches produce a fuzzier, noisier one. That noisiness is not instability — a jittery small-batch curve can be perfectly healthy — so do not confuse a thick noise band with the jagged spikes of a too-high LR.
Second, batch size interacts with the effective learning rate. A common heuristic scales the rate with batch size (linear or square-root scaling), so changing the batch without adjusting the rate shifts the curve’s descent speed. Larger batches also reach a point of diminishing returns — beyond a ‘critical batch size’ you spend more samples per step for little extra progress per step, so the loss-versus-tokens curve stops improving even as the loss-versus-steps curve looks tidy. When you compare two runs, always fix the x-axis you care about: loss-versus-steps flatters big batches, while loss-versus-tokens (or versus wall-clock, or versus FLOPs) is what actually tells you which configuration is more efficient.
Gradient norm: the companion signal that moves first
Almost every diagnosis above leaned on the global gradient norm, and that is deliberate: it is the single most valuable companion to the loss because it often moves before the loss does. The norm is the length of the flattened gradient vector across all parameters, ||g|| = sqrt(Σ g_i^2), and its trajectory tells you what the optimizer is experiencing, not just the outcome.
Read it as a second track beside the loss. A stable, bounded norm is the healthy default. A norm spiking to many times its baseline is the leading indicator of an impending loss spike or divergence — you often see the norm jump a step or two before the loss reacts, which is exactly why skip-step logic watches the norm. A norm collapsing toward zero while the loss is flat distinguishes a dead stall (nothing left to learn, or a vanished LR) from a productive plateau (norm still active). And a norm that is chronically tiny from early on points at a too-low LR or a vanishing-gradient problem in the architecture. Logging the gradient norm costs almost nothing and roughly doubles the diagnostic power of your dashboard; treating loss and gradient norm as a pair, rather than watching loss alone, is the habit that most separates fast debugging from slow.
A numeric-cue cheat sheet and a debugging order
Diagnosis is faster when you have reference numbers in your head rather than reasoning from scratch each time. The table collects the cues used throughout this article into checkable form.
| Observation | Numeric cue | Likely diagnosis |
|---|---|---|
| Step-0 loss | ≈ ln(vocab) (e.g. 10.82 at V=50257) | Far below → label leak; far above → bad init |
| Curve jagged, recovers | grad-norm spikes 10–100× baseline | LR slightly high; tighten clip (~1.0), lengthen warmup |
| Dips then climbs to NaN | grad-norm → inf; fp16 no loss-scale | LR too high / precision; cut LR, use bf16 |
| Smooth but too high & flat | grad-norm small; loss ~1 unit above target | LR too low; raise it, run an LR range test |
| Train down, val up | val minimum then rises | Overfit (fine-tune/multi-epoch); early-stop at trough |
| Both high, small gap | grad-norm quiet | Underfit; add capacity / train longer |
| Flat, then resumes | grad-norm still non-zero | Tiered plateau; wait, do not kill |
| Warmup ~500–2000 steps | a few % of total steps | Early wobble → too short; slow start → too long |
And a working order when a curve looks wrong: (1) check the start height against ln(V) — rule out data/init bugs first; (2) overlay the gradient norm to see whether the problem is explosive, stalled, or quiet; (3) match the shape to the gallery — jagged/divergent means LR down, flat/high means LR up, val-split means regularize or stop; (4) change one knob and re-run a short probe rather than several at once, so the next curve is interpretable. Reading loss curves is not mysticism — it is a small, learnable set of shapes, each anchored to a number you can check and a knob you can turn.