Tokenizer Math — BPE and Subword

Why not just use characters or words?

The tokenizer’s job is to pick the unit the model operates on, and the two obvious choices both fail. Character-level tokenization gives a tiny vocabulary (a few hundred symbols) and never meets an unknown token, but it makes sequences brutally long: a 1,000-word document becomes ~5,000–6,000 characters, so attention — which is O(N^2) in sequence length — pays enormously, and the model must relearn spelling from scratch before it can reason about meaning.

Word-level tokenization gives short sequences and meaning-bearing units, but the vocabulary explodes: real corpora contain hundreds of thousands of distinct words, plus every typo, inflection, URL, and proper noun. Any word not in the fixed vocabulary collapses to a single <UNK> token, destroying information, and morphology (run, running, runner) is scattered across unrelated IDs. Subword tokenization is the engineered compromise: keep a bounded vocabulary (say 32k–128k) of frequent whole words and reusable fragments (ing, tion, pre), so common words stay one token, rare words decompose into known pieces, and nothing is ever truly out-of-vocabulary. That single design choice is what the rest of this article makes quantitative.

A token is a row in the embedding table

Before the math, fix the mental model. The tokenizer owns a vocabulary: a bijection between token strings and integer IDs 0 … V-1. Encoding a prompt produces a list of those integers; the very first thing the model does is use each integer as a row index into an embedding matrix E of shape [V, d_model], pulling out a d_model-dimensional vector per token.

So a token is not an abstract idea — it is literally one learnable row of E. This is why vocabulary size is not free: every entry you add is another d_model parameters the model must store and learn, and a token that appears rarely in training gets a poorly-estimated row. It is also why the tokenizer must be frozen for the life of a model: the IDs are hard-wired to embedding rows, so you cannot swap tokenizers without retraining the embedding (and usually the whole model). Two properties matter for what follows: the vocabulary is fixed and finite, and each token carries a fixed per-token cost in parameters, in sequence position, and — as we’ll see — in dollars.

Byte-Pair Encoding: the core idea

Byte-Pair Encoding (BPE) is a compression algorithm repurposed for tokenization. It builds the vocabulary bottom-up and greedily: start from the smallest possible alphabet, then repeatedly glue together whichever adjacent pair is most frequent, promoting that pair to a new token. The intuition is pure information theory — the pairs that recur most often are the ones worth spending a vocabulary slot on, because collapsing them shortens the corpus the most.

vocab ← all base symbols (e.g. 256 bytes)
merges ← []                       # ordered list of learned rules
represent every word as a sequence of base symbols
while len(vocab) < target_size:
    counts ← frequency of every adjacent symbol pair in the corpus
    (a, b) ← argmax(counts)        # most frequent pair
    vocab.add(a + b)                    # new token
    merges.append((a, b))               # remember the rule AND its rank
    replace every adjacent (a, b) with the merged symbol a+b

Two outputs come out of training: the vocabulary (the set of tokens) and the ordered merge list (the rules, in the order learned). The order is not bookkeeping — it is the algorithm at inference time, because merges are replayed in exactly the rank they were learned. Training is offline and done once; the learned tables ship with the model.

A worked toy example, traced by hand

Take a tiny corpus of five word-types with their frequencies: hug ×10, pug ×5, pun ×12, bun ×4, hugs ×5. The base alphabet is the characters that appear: b g h n p s u. Split every word into characters and count adjacent pairs, weighted by word frequency:

(u,g): 10+5+5 = 20   ← winner
(p,u): 5+12    = 17
(u,n): 12+4    = 16
(h,u): 10+5    = 15
merge (u,g) → "ug"

Now the words are h·ug, p·ug, p·u·n, b·u·n, h·ug·s. Recount and merge the new winner:

merge 2:  (u,n): 12+4 = 16   → "un"
merge 3:  (h,ug): 10+5 = 15  → "hug"

After three merges the vocabulary is {b, g, h, n, p, s, u, ug, un, hug} and the ordered merge list is (u,g), (u,n), (h,ug). Notice what BPE discovered on its own: ug and un are reusable subword units, and hug earned a whole-word token because it was common. Nobody told it about morphology — frequency alone carved the corpus at useful joints.

Encoding at runtime: replay the merges

Training learned the rules; encoding applies them. A new string is first broken into base symbols, then the merge list is replayed greedily, always applying the lowest-rank (earliest-learned) applicable merge until none remain:

def encode(word, merges):
    syms = list(word)                       # base symbols
    while True:
        pair = lowest_rank_adjacent_pair(syms, merges)
        if pair is None: break              # no rule applies
        syms = apply(syms, pair)            # merge it, repeat
    return [vocab_id[s] for s in syms]

Trace bug against the toy model: split to b·u·g; rule (u,g) applies → b·ug; no further rule matches, so bug becomes two tokens [b, ug]. A word with no learned merges, or containing a symbol outside the base alphabet, simply stays split into its smallest pieces — which is why a byte-level base alphabet (next section) guarantees nothing ever fails to encode. Because encoding is deterministic and depends only on the frozen tables, the same string always yields the same IDs. Production tokenizers like tiktoken implement exactly this loop in optimized C++/Rust; the per-document cost is small but non-zero, which matters when you tokenize streams on the fly.

Byte-level BPE: an alphabet that never fails

What is the base alphabet? The elegant modern answer, introduced by GPT-2, is the 256 raw bytes. Instead of treating text as Unicode characters (of which there are ~150,000, an awkward and unbounded base), treat it as a stream of bytes. Every possible string — any language, emoji, control character, malformed UTF-8, binary blob — is by construction a sequence of bytes, so there is exactly one starting alphabet of size 256 and there is no such thing as an out-of-vocabulary input. BPE merges then build familiar subwords on top of the byte layer.

This is byte-level BPE (BBPE), and it is what GPT, Llama, and Phi use. The cost is that a single exotic Unicode character can span several bytes and therefore several tokens before any merges apply — one reason non-Latin scripts are token-expensive. The alternative, Unicode-level modeling (as in classic SentencePiece configurations), starts from characters and handles the unknown-symbol problem differently. The edge cases differ, but the headline is the same: a byte base alphabet trades a slightly higher token count on rare scripts for the guarantee that anything can be encoded losslessly and round-tripped back to the exact original bytes.

Vocabulary size is an embedding-table budget

Here is where tokenizer choices become model parameters. The token embedding matrix has shape [V, d_model], so it holds exactly V × d_model parameters. Most models also have an output (un-embedding) projection of the same shape mapping the final hidden state to logits over the vocabulary; when the two are not weight-tied, you pay for both. So the parameter cost of the vocabulary is:

embedding params  = V × d_model
unembedding params = V × d_model   (0 if tied)
vocab params (untied) = 2 × V × d_model

This scales linearly and only with V and d_model — it is independent of the number of transformer layers, because embeddings live at the model’s edges, not in the stack. That has a subtle consequence: for a small model the embedding table can be a startling fraction of the total parameters, while for a large model it is a rounding error. A generous vocabulary is therefore relatively cheaper — as a fraction of the whole — the bigger the model gets, which is part of why frontier models have drifted from ~32k toward ~128k+ vocabularies. The next section puts real numbers on it.

A vocab/parameter numeric example

Take d_model = 4096 (a typical ~7B-class hidden size) and compare two vocabularies:

V = 32,000  → 32,000 × 4,096  = 131,072,000   ≈ 131M params (one matrix)
V = 128,000 → 128,000 × 4,096 = 524,288,000  ≈ 524M params (one matrix)

untied (embed + unembed):
V = 32,000   → ≈ 262M params
V = 128,000  → ≈ 1.05B params

Quadrupling the vocabulary from 32k to 128k costs an extra ~393M parameters per matrix — nearly 0.8B if untied. In a 7B model that is roughly 11% of all parameters going to the un-embedding pair; in a 70B model the same absolute cost is only ~1.5%. So the trade is real but shrinks with scale. What do you get for it? A bigger vocabulary means more whole words and longer fragments live in the table, so text encodes into fewer tokens — better compression. You are literally spending parameters (and embedding memory) to buy shorter sequences. Whether that is worth it depends on the numbers in the next few sections: shorter sequences mean cheaper attention, smaller KV cache, and lower per-request token bills.

The core tension: sequence length vs vocabulary

Vocabulary size sits at the center of a genuine tug-of-war, and both directions have real costs. Push the vocabulary up and each piece of text compresses into fewer tokens: sequences get shorter, attention (O(N^2)) and the KV cache (linear in N) get cheaper, and you fit more real content in a fixed context window — but the embedding table grows, and the rarest tokens are seen so seldom in training that their rows are poorly learned.

Push the vocabulary down and the embedding table shrinks and every token is well-trained — but text shatters into more, smaller tokens, so sequences lengthen, attention costs rise, and your context window holds less meaning. There is no universally correct answer; the sweet spot depends on model size (large models amortize a big table easily), the target languages (multilingual models need bigger vocabularies to serve many scripts fairly), and the deployment budget. The industry has converged on roughly 32k–50k for earlier and smaller English-centric models and 100k–256k for recent large and multilingual ones — a drift upward that exactly tracks the ‘bigger models make big vocabularies cheap’ logic above.

What happens to rare and unseen tokens

Subword tokenization’s quiet superpower is graceful degradation. There is no <UNK>. A word the tokenizer has never seen as a whole — a novel product name, a rare surname, a typo, a chemical formula — is simply decomposed into the largest known fragments, falling back all the way to individual bytes if necessary. The string is always representable and always reversible; nothing is lost.

The price is paid in length, not in information. A familiar word like tokenizer might be one or two tokens; an unusual string like antidisestablishmentarianism or Kryptonite-9000 fractures into many. This is visible and exploitable: made-up words, long hashes, base64 blobs, and dense code all have poor compression and therefore high token counts. It also has a training-quality dimension — a token that appears only a handful of times in the training corpus (a so-called ‘glitch’ or under-trained token) has a badly estimated embedding row and can trigger strange model behavior. The practical takeaway: if a particular vocabulary (a domain jargon, a programming language, a natural language) matters to you, check how your real text tokenizes rather than trusting an average.

Compression ratio: chars per token, tokens per word

The single most useful tokenizer metric is its compression ratio — how much text one token carries. Two equivalent framings dominate:

chars_per_token = total_characters / total_tokens
tokens_per_word = total_tokens / total_words

For modern English BPE tokenizers, the rules of thumb are remarkably stable: about ~4 characters per token and roughly ~1.3 tokens per word (equivalently, ~100 tokens per ~75 words). So a token is, on average, a bit less than a short English word — often a word, sometimes a sub-word chunk of a longer one. Concretely, a 750-word essay is on the order of 1,000 tokens; a 100,000-token context window holds roughly 75,000 English words, or about 150 pages.

These are averages over English prose, and the variance matters. Code, with its punctuation and identifiers, compresses worse (more tokens per character); repetitive or highly templated text compresses better. The number is not a property of the language alone — it is a property of the tokenizer applied to that text, which is exactly why the same sentence can have very different token counts under different models.

Compression drives context length and cost

Compression ratio is not a curiosity — it is the exchange rate between text and everything you pay for. A context window is measured in tokens, so better compression directly enlarges the effective window: at 4 chars/token a 128k-token window holds ~512k characters, but a tokenizer that manages only 3 chars/token on your text fits just ~384k characters in the same window — a 25% haircut with no change to the model.

The same lever moves compute and money. Attention scales as O(N^2) and the feed-forward layers and KV cache scale as O(N) in the token count N, so fewer tokens for the same content means less compute, lower latency, and a smaller memory footprint per request. And because commercial APIs bill per token — input and output separately — the compression ratio is quite literally a price multiplier on your bill. Two providers with identical per-token prices are not equally expensive if one’s tokenizer needs 10% more tokens to represent your workload. When comparing models, the honest unit is cost per unit of text (per character or per word), not cost per token.

Fertility: why some languages cost 3x more

Fertility is the compression ratio viewed per language: the average number of tokens the tokenizer emits per word (or per character) of that language. It is where tokenizer economics become a fairness issue. Most widely used tokenizers are trained on corpora dominated by English and other high-resource Latin-script languages, so BPE spends its merge budget learning English subwords. Those languages end up with low fertility — efficient, ~1.3 tokens/word.

Languages the tokenizer saw less of, or that use non-Latin scripts, fare far worse. Morphologically rich languages (Finnish, Turkish, Tamil) pack meaning into long inflected words that BPE never learned as units, so they fragment; and scripts outside the merge-rich Latin range (Devanagari, Tamil, Chinese, Arabic) can burn several bytes, hence several tokens, per character under byte-level BPE. The result: the identical sentence, with the identical meaning, can cost 2–4× more tokens in Hindi or Tamil than in English. That surcharge compounds through everything downstream — a user in a high-fertility language pays more per API call, waits longer, and gets a smaller effective context window, because their words consume more of the fixed token budget. Multilingual models fight this with larger, more balanced vocabularies, which is a major reason vocabulary sizes have grown.

Token count is the unit of compute and price

Pull the threads together and one quantity governs the whole economic and performance story: the number of tokens. It is the input to the model’s cost function in three simultaneous senses. First, compute: a forward pass does O(N) work in the feed-forward and projection layers and O(N^2) in attention, so halving the tokens for a given piece of text more than halves the attention cost. Second, memory: the KV cache that makes generation efficient grows linearly with token count, so token count caps how much context and how many concurrent requests fit in memory. Third, price: APIs meter input and output tokens directly.

This is why the tokenizer — a component most people never think about — silently sets the ceiling on cost, latency, and capacity. Every optimization at the tokenizer level (a better-matched vocabulary, a domain-tuned tokenizer, prompt phrasing that compresses well) pays off everywhere at once. And it reframes prompt engineering economically: a verbose prompt is not just stylistically heavier, it is more tokens — more compute, more memory, more money — on every single call. The cheapest token is the one you never emit.

The tokenizer siblings: BPE, WordPiece, Unigram, SentencePiece

BPE is one member of a small family, and it helps to place the others. All share the subword goal; they differ in how they choose the vocabulary and how they segment.

The distinctions are real but narrower than the names suggest. BPE and WordPiece are both greedy bottom-up mergers differing mainly in the merge criterion (frequency vs likelihood gain). Unigram inverts the direction — it starts large and prunes, and it can represent multiple valid segmentations probabilistically. SentencePiece is orthogonal: it is the tooling that makes any of these operate directly on raw Unicode text with whitespace treated as a normal symbol, which is what makes it clean for languages that don’t delimit words with spaces. In practice a modern ‘SentencePiece-BPE’ tokenizer (as in Llama) is BPE’s merge algorithm running inside SentencePiece’s raw-text framework — the same math you traced by hand above, industrialized.

Pitfalls and what to actually measure

A few traps recur once you take tokenizers seriously. Do not estimate token counts from word counts across languages — the ~1.3 tokens/word rule is English-only; fertility can triple it. Do not assume whitespace is free: in byte-level and SentencePiece tokenizers a leading space is part of the token, so " the" and "the" are different tokens, and trailing spaces or double newlines quietly cost tokens. Do not trust averages for code or structured data — JSON, indentation, and long identifiers compress poorly and can blow past a naive estimate.

The constructive version: measure on your data. Run your real prompts and documents through the exact tokenizer your model uses and record chars_per_token and tokens_per_word for the languages and formats you actually serve. Budget context windows in the resulting tokens, not in characters or words. Compare vendors on cost per unit of text, not per token. And remember the through-line of this whole article: the tokenizer is a small deterministic function that fixes three numbers — embedding parameters, sequence length, and per-request token cost — that then propagate into every dimension of a model’s economics. It is worth understanding precisely, because it is paying rent on every single call.