Perplexity is the number you watch while a language model trains, and it is deceptively simple: it is just the exponential of the cross-entropy loss. But packed inside that one transformation is a whole theory of what ‘a good language model’ means — a link to information theory, to data compression, and to the effective number of choices a model faces at every token. It is also one of the most misused numbers in the field, because a perplexity of 12 for one model and 8 for another can mean nothing at all if the two used different tokenizers or different test text. This piece builds perplexity from the ground up: the cross-entropy underneath it, the branching-factor intuition, a fully worked numeric example, the nats-versus-bits subtlety, the bits-per-character and bits-per-byte variants that make cross-model comparison honest, the compression connection, and finally why a low perplexity is necessary but never sufficient — and where downstream benchmarks have to take over.
What perplexity actually measures
Perplexity answers one question: on average, how surprised is the model by the next token? A language model is a probability distribution — given the tokens seen so far, it outputs a probability for every possible next token. If the true next token was one the model rated highly, the model was not surprised; if the true token was one it rated near zero, it was very surprised. Perplexity aggregates that surprise across a whole test set into a single number.
The intuition to hold onto is that perplexity is an effective branching factor: it reports roughly how many equally-likely options the model was choosing between at each step. A perplexity of 1 means perfect prediction (the model assigned probability 1 to the right token every time — no uncertainty). A perplexity of 50,000 means the model is as lost as if it were guessing uniformly among fifty thousand tokens. Real language models on English text land somewhere in the single or low double digits — the model has narrowed a vocabulary of tens of thousands down to an effective handful of live candidates at each position. Lower is better, always.
Cross-entropy: the loss underneath
Perplexity is built directly on the cross-entropy loss, which is what a language model is actually trained to minimize. For a test sequence of N tokens, with the model assigning probability p_i = P(token_i | token_1 ... token_{i-1}) to each true token, the cross-entropy (in nats) is the average negative log-likelihood:
H = -(1/N) Σ_i ln(p_i)
# p_i = probability the model gave the ACTUAL next token at position i
# ln = natural log, so H is measured in 'nats'
# H is exactly the training loss, averaged over the test setEach term -ln(p_i) is the surprise of one token: it is 0 when p_i = 1 (no surprise) and grows without bound as p_i → 0 (a confident wrong prediction is punished severely). Averaging over all N tokens gives the mean surprise per token. This is the quantity gradient descent drives down; perplexity is simply a more interpretable way of reporting the very same thing, rescaled so the units are ‘effective choices’ rather than ‘average log-loss.’
Perplexity = exp(cross-entropy)
The definition is one line, but the log base matters, so state it every time. When cross-entropy H is measured in nats (natural log), perplexity is:
perplexity = exp(H) = exp( -(1/N) Σ_i ln(p_i) )
# equivalently, the inverse geometric mean of the true-token probabilities:
perplexity = ( ∏_i p_i )^(-1/N)
# if H is measured in BITS (log base 2) instead, use 2^H, not exp(H)The two forms are identical: exponentiating an average of logs turns the sum into a product and the average into an N-th root, giving the geometric mean of 1/p_i. That geometric-mean view is worth internalizing — perplexity is the reciprocal of the typical probability the model assigns to the right token. If the model typically gives the true token probability 1/10, perplexity is about 10. The exponential is what converts the additive world of log-loss (nice for optimization) back into the multiplicative world of probabilities and counts (nice for intuition). Get the base wrong — exp on a bits value, or 2^ on a nats value — and your perplexity is silently, badly wrong.
The branching-factor intuition
Why call perplexity an ‘effective number of choices’? Consider the simplest possible model: one that assigns uniform probability 1/V to every token in a vocabulary of size V. Every true token then has p_i = 1/V, so H = -ln(1/V) = ln(V) nats, and perplexity = exp(ln V) = V. A model that has learned nothing beyond the vocabulary size has perplexity exactly V — it is genuinely choosing uniformly among all V tokens.
Now a real model does better than uniform, and perplexity measures how much better in the same units. A perplexity of 20 means the model’s uncertainty at each token is equivalent to a uniform choice among 20 options — even though the vocabulary might be 50,000, the model has effectively ruled out all but ~20 live candidates on average. This is why perplexity is so intuitive: it collapses a complicated, position-varying probability distribution into a single ‘as if it were a fair die with this many sides’ number. The catch, which the next sections unpack, is that the number of ‘sides’ depends heavily on what a ‘token’ is.
A fully worked computation
Concrete numbers make it stick. Suppose we evaluate a model on a tiny 5-token sequence, and the probabilities it assigned to the actual next tokens were: 0.5, 0.25, 0.1, 0.4, 0.2. Compute the cross-entropy in nats, then perplexity:
surprises (nats): -ln(0.5)=0.6931 -ln(0.25)=1.3863 -ln(0.1)=2.3026
-ln(0.4)=0.9163 -ln(0.2)=1.6094
sum = 6.9077 nats over 5 tokens
H = 6.9077 / 5 = 1.3815 nats/token
PPL = exp(1.3815) = 3.98
# cross-check via the geometric-mean form:
product = 0.5 * 0.25 * 0.1 * 0.4 * 0.2 = 0.001 = 10^-3
PPL = (0.001)^(-1/5) = 10^(3/5) = 10^0.6 = 3.98 # same answerBoth routes agree: perplexity ≈ 3.98. The reading is that across these five predictions the model was, on average, as uncertain as if it were guessing among about four equally-likely tokens. Notice how the low-probability token (0.1, surprise 2.30 nats) dominates the sum — perplexity is punishing, and geometric by nature, so a single confidently-wrong token drags the whole score up far more than a confidently-right token pulls it down. That asymmetry is exactly why models learn to hedge rather than commit.
Nats, bits, and the exp-vs-2 convention
Cross-entropy is a log, and the base of that log is a unit choice, not a change of meaning. Natural log gives nats; log base 2 gives bits (also called shannons). They convert by a constant: bits = nats / ln(2), with ln(2) ≈ 0.6931. Perplexity itself is unit-free — it comes out the same either way — provided you match the exponential to the base:
H_nats = 1.3815 → PPL = exp(H_nats) = e^1.3815 = 3.98
H_bits = 1.9932 → PPL = 2^(H_bits) = 2^1.9932 = 3.98
# 1.9932 bits = 1.3815 nats / 0.6931 . Same information, different ruler.Frameworks report the training loss in nats (it is just the mean negative log-likelihood with a natural log), so perplexity = exp(loss) is the everyday formula. Information-theory and compression discussions prefer bits, because a bit is a physically meaningful unit — one yes/no answer. When you see ‘1.99 bits per token’ and ‘perplexity 3.98’ side by side, they are the same measurement in two languages. The only real error mode is crossing the wires: applying exp() to a bits value inflates perplexity, and applying 2^ to a nats value deflates it.
Bits-per-character and bits-per-byte
Per-token perplexity has a hidden dependency — the token — so the fairest cross-model metrics normalize to a unit that every model shares: the character, or better, the raw byte. Bits-per-character (BPC) and bits-per-byte (BPB) take the model’s total surprise over a passage, in bits, and divide not by the number of tokens but by the number of characters or UTF-8 bytes in the original text:
total_bits = -Σ_i log2(p_i) # summed over all tokens
BPB = total_bits / (number of BYTES in the raw text)
BPC = total_bits / (number of CHARACTERS in the raw text)
# tokens cancel out: a model with more, smaller tokens has more terms
# in the sum but the same total bits for the same underlying textBecause bytes and characters are properties of the text, not of the tokenizer, BPB and BPC are directly comparable across models with different vocabularies. This is why serious language-modeling benchmarks — enwik8, enwik9, text8 — are scored in bits-per-character or bits-per-byte, not perplexity. A state-of-the-art model on enwik8 reports something like ~0.95 BPC; that number means something concrete and portable in a way that ‘perplexity 8’ does not.
Why perplexity depends on the tokenizer
This is the single most important caveat, and it sinks more naive comparisons than any other. Perplexity is defined per token, and different models tokenize the same text into different numbers of tokens. A tokenizer that splits text into many short pieces makes each individual next-token prediction easier — after in and ter, predicting national is nearly certain — so the per-token perplexity drops, without the model being any better at modeling language.
Concretely, imagine two models of identical quality on the same text. Model A uses a byte-level vocabulary and needs 6 tokens for a word; Model B uses a word-level vocabulary and needs 1. They assign the same total probability to the word, but Model A spreads that surprise over 6 tokens and Model B concentrates it into 1 — so Model A shows a much lower per-token perplexity purely as an artifact of tokenization. Comparing their raw perplexities would falsely crown Model A. The fixes: compare perplexity only between models sharing a tokenizer and test set, or convert to bits-per-byte, where the token count cancels and the comparison becomes honest. Cross-model perplexity without this care is meaningless.
Perplexity and compression are the same thing
There is a beautiful identity hiding here: a language model is a compressor, and its cross-entropy is the compressed size. Shannon’s source coding theorem says the shortest expected code length for a symbol drawn from a distribution is the entropy of that distribution, in bits. An arithmetic coder paired with the model’s predicted probabilities achieves almost exactly this: it encodes each token in -log2(p_i) bits.
So the total bits the model spends — the same sum that defines cross-entropy — is literally the size, in bits, of the text after compressing it with the model. That makes bits-per-byte the compression ratio: a model at 0.95 BPB compresses text to 0.95 bits for every 8-bit byte, roughly an 8x reduction, and beats general-purpose compressors like gzip precisely because it predicts language better. Lower perplexity means better compression, exactly and quantitatively. This is the basis of the ‘compression is intelligence’ framing and of benchmarks like the Hutter Prize, which reward compressing Wikipedia as a proxy for language understanding. When you minimize cross-entropy, you are training the world’s best text compressor for your data.
Per-token, per-word, and normalization traps
Because the normalizer is a choice, you must know which one a reported number uses. Three conventions circulate. Per-token perplexity divides by the token count — the training-loss default, but tokenizer-dependent. Per-word perplexity divides the total surprise by the number of words instead, which is how the classic n-gram and WikiText literature reports it; it partially removes the tokenizer’s influence but depends on a word-splitting rule. Per-character / per-byte (BPC/BPB) divides by characters or bytes and is the most portable.
A subtle trap: word-level perplexity and token-level perplexity for the same model on the same text are different numbers, related by the average number of tokens per word. If a model averages 1.3 tokens per word, its per-word perplexity is roughly its per-token perplexity raised to the 1.3 power — noticeably higher. Two more quiet gotchas: how you count the first token (which has no context) and how you handle sequences longer than the context window (sliding-window evaluation with overlap versus non-overlapping chunks) can each shift the number by a few percent. Always report the normalizer, the test set, and the evaluation stride.
Reading perplexity numbers in the wild
What counts as a ‘good’ perplexity is entirely relative to the text and tokenizer, so anchor every number to its dataset. As a rough, illustrative guide on standard English benchmarks like WikiText-103, older and smaller models sit higher (a GPT-2-class model in the tens) while larger modern models push into the single digits — but these are not leaderboard-comparable across tokenizers, and quoting a bare ‘GPT-4 has perplexity 5’ is exactly the mistake this article warns against.
The useful discipline is relative, not absolute. Track your own model’s perplexity on a fixed held-out set across training steps and architecture changes: a drop from 25 to 18 on the same data with the same tokenizer is real progress. Compare two of your own models only if they share the tokenizer and evaluation text. And treat any cross-paper perplexity comparison with suspicion unless the authors have pinned down the tokenizer, the dataset, the normalization, and the sliding-window stride. Perplexity is a superb relative instrument and a treacherous absolute one.
The limits of perplexity
Perplexity measures one thing well — average next-token likelihood on held-out text — and is blind to almost everything else you care about. It rewards a model for being fluent and calibrated on the distribution of the test corpus, but a low perplexity does not imply the model can reason, follow instructions, avoid hallucination, do arithmetic, or write working code.
Several gaps are structural. First, perplexity is dominated by the easy, high-frequency tokens (function words, punctuation, boilerplate); the rare tokens where reasoning actually lives contribute little to the average. A model can shave perplexity by getting commas and the ever more right while its factual accuracy stagnates. Second, perplexity is a base-model metric — it evaluates raw next-token prediction, not the instruction-following or chat behavior that fine-tuning and RLHF install, and those stages can even raise perplexity on generic text while making the model far more useful. Third, it says nothing about generation quality under sampling, long-range coherence, or safety. Perplexity is necessary — a model with terrible perplexity will be terrible — but nowhere near sufficient. It is the thermometer, not the diagnosis.
Downstream benchmarks: what perplexity can't see
Because perplexity misses capability, the field leans on downstream benchmarks that score the model on concrete tasks. Each probes a different axis: MMLU for broad multiple-choice knowledge across dozens of subjects, GSM8K for grade-school math word problems, HumanEval for code-generation correctness (does the function pass its tests?), HellaSwag and LAMBADA for commonsense completion and long-range word prediction, and MT-Bench or arena-style human preference for open-ended chat quality.
These correlate with perplexity only loosely and only within a family: driving perplexity down usually helps, but two models at the same perplexity can differ wildly on MMLU because of what data they saw and how they were tuned. The honest evaluation stack uses perplexity as the cheap, high-frequency signal during pretraining — you can compute it every few hundred steps — and reserves the expensive benchmark suite for milestones, since running MMLU or HumanEval means generating and grading thousands of completions. Perplexity tells you the model is learning the distribution; benchmarks tell you whether that translated into anything a user would value.
Perplexity for CPU-scale small models
For the small language models this series targets — trained from scratch on a single GPU or even a CPU — perplexity is the workhorse metric, and the big-model benchmarks are mostly the wrong tool. A model with a few tens of millions of parameters simply lacks the capability floor that MMLU or GSM8K assume; it will score at or near chance, giving you no gradient of progress to steer by. The benchmark scores are flat noise while your model is, in fact, learning.
Perplexity, by contrast, moves smoothly and informatively the entire way down. Watch it on a fixed held-out split of your training distribution and you get a dense, cheap, low-variance signal after every checkpoint. When you do want a capability check at this scale, choose the easiest benchmarks — LAMBADA (last-word prediction) and HellaSwag (commonsense completion) can register above-chance signal from a small model, whereas knowledge-heavy suites cannot. And if you are comparing tokenizer or vocabulary choices for your SLM, switch to bits-per-byte so the comparison is not silently rigged by token granularity. For CPU-SLM work: track perplexity (or BPB) constantly, sanity-check with the smallest benchmarks occasionally, and ignore the leaderboard giants.
A practical evaluation checklist
Pulling it together into habits that keep perplexity honest and useful. First, fix your evaluation set and freeze it — a held-out sample of your training distribution, never touched by training, identical across every checkpoint, so trends are comparable. Second, always report the base and normalizer: state whether it is per-token, per-word, or per-byte, and whether the loss was in nats or bits, so exp versus 2^ is unambiguous.
Third, never compare perplexity across tokenizers; convert to bits-per-byte when the vocabularies differ. Fourth, pin the evaluation stride for long documents — sliding-window with overlap gives a truer number than disjoint chunks, but only if you apply it consistently. Fifth, use perplexity for the high-frequency signal and benchmarks for the milestones: perplexity to steer daily, task benchmarks to validate that capability actually emerged. Finally, remember what the number cannot tell you — a beautiful perplexity curve is fully compatible with a model that hallucinates, cannot follow instructions, and fails every reasoning task. Measure perplexity because it is the cheapest true signal you have, and distrust it because it is only ever half the picture.