Tokenizers Decide Your Bill: How BPE Merges Treat Languages and Code

From bytes to tokens: how BPE works

A byte-level BPE tokenizer starts from the UTF-8 bytes of the text, so its base vocabulary is the 256 possible byte values. Any string can be encoded, which is why byte-level tokenizers never produce an unknown token. Training then repeats one step: count every adjacent pair of symbols in the training corpus, take the most frequent pair, add it to the vocabulary as a new symbol, and replace every occurrence. After m merges the vocabulary has 256 + m entries (plus any special tokens). GPT-2 used 50,000 merges, for a vocabulary of 50,257 including its end-of-text token; recent models use vocabularies in the hundreds of thousands.

Encoding new text replays the learned merges in the order they were learned: within each chunk of text, repeatedly apply the earliest-learned merge that matches an adjacent pair, until no learned merge applies. The consequence is the whole subject of this article: text that looks like the training data compresses into few tokens, and text that does not falls back toward one token per byte.

"the"  -> bytes 74 68 65 -> after merges (t,h) and (th,e): 1 token
" Store" with a leading space: frequent in training -> 1 token
"बेलगावी" (7 Devanagari code points, 21 bytes) never seen -> up to 21 tokens

The original BPE work for neural machine translation used characters as the base alphabet; GPT-2 moved to bytes so that every Unicode string is representable. SentencePiece offers BPE and unigram language model variants, treats whitespace as an ordinary symbol, and can use byte fallback for characters outside its learned vocabulary. The mechanics differ in detail, but all of them share the same economics: frequency in the training data buys compression.

The arithmetic of scripts

UTF-8 uses one byte for ASCII, two bytes for many Latin-extended, Greek, Cyrillic, Hebrew and Arabic characters, three bytes for most other scripts in the Basic Multilingual Plane (including Devanagari, Tamil, Thai and CJK ideographs), and four bytes for emoji and supplementary characters. Before any merges, an English sentence costs about one token per character, while a Hindi sentence costs about three.

Merges can close that gap only if the tokenizer saw enough of the script during training. With a training mix that is mostly English and code, almost every merge is spent on English words and code fragments. Hindi words remain sequences of bytes, often not even aligned to characters, because a merge that would join the three bytes of one Devanagari letter never became frequent enough to be learned.

The second row is the surprising one: training merges makes the gap wider, because English gets cheaper and Hindi does not. Studies of commercial and open tokenizers have measured large differences of exactly this kind across languages, in some cases more than ten times for the same content. The cost is not only money: a context window holds proportionally less text, rate limits expressed in tokens per minute admit fewer requests, and generated answers take longer to stream because output is produced one token at a time.

The pre-tokenizer regex: a hidden second policy

Before BPE sees anything, most byte-level tokenizers split the text into chunks with a regular expression, and merges never cross chunk boundaries. GPT-2's pattern is roughly:

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

This keeps a leading space attached to the following word, separates letters from numbers and punctuation, and handles runs of whitespace. It was designed around languages where a word is a run of letters. In Devanagari and many other Indic scripts, dependent vowel signs (matras) and the virama are combining marks, Unicode category M, not letters. A pattern that matches \p{L}+ therefore ends a chunk at every vowel sign and starts a new punctuation-class chunk for the mark itself. The word किताब (book) becomes several chunks, and no amount of training data can produce a single token for it, because BPE never sees the whole word as one sequence.

Including combining marks in the letter class (for example [\p{L}\p{M}]+) lets a Hindi word stay in one chunk. Some newer tokenizers do include marks in their letter classes; check the actual pattern of the tokenizer you use rather than assuming. With Hugging Face tokenizers it is in the pre_tokenizer section of tokenizer.json; with tiktoken it is the pattern string of the encoding. Mission two of the lab puts you in exactly this trap: Hindi is in the training data, the ratio is still about 2.5×, and the fix is the regex, not more data.

Code, whitespace and identifiers

Code has its own token economics. Indentation is a run of spaces, and in a byte-level tokenizer each space is a byte. If the training data contained little code, a 12-space indent costs many tokens, and deeply nested code pays that on every line. Identifiers split at underscores and case changes into pieces, and operators like += or ): compress only if they were frequent in training. Tokenizers built for code-heavy models typically add tokens for runs of whitespace and learn common identifier fragments, which is why the same source file can differ substantially in token count between tokenizers.

In the lab, the deeply nested code event switches production code to six levels of indentation. With no code in the tokenizer's training mix, code costs about 780 tokens per 1,000 characters; with 10% code it drops to about 340, because indentation runs and common fragments become single tokens. Every percent of training mix given to code is taken from something else, though, which is why mission three asks you to fix code without making English or Hindi worse. For production, the practical lessons are simpler: strip or normalise indentation where semantics allow (minified JSON, collapsed whitespace in retrieved snippets), avoid sending unchanged boilerplate on every turn, and measure token counts on your real payloads.

Vocabulary size is a trade-off, not a free lunch

More merges compress text further, but each new token adds a row to the embedding matrix and a column to the output projection. The parameter cost is vocabulary × hidden size, once for input embeddings and once more if the output layer is not tied:

vocab 50,000  x d_model 4,096 = 205 M parameters (about 0.41 GB in bf16)
vocab 200,000 x d_model 4,096 = 819 M parameters (about 1.64 GB in bf16)
# double these if input and output embeddings are untied

The output softmax over the vocabulary also costs compute at every generated token. And diminishing returns set in: later merges join rarer pairs, so each saves fewer tokens across real traffic. Rare tokens also get fewer training updates, which can leave their embeddings poorly trained. The right vocabulary is the smallest one that compresses your traffic mix well, which is the question mission four asks with a hypothetical traffic shift. In the lab the synthetic corpus runs out of frequent pairs after a little under 900 merges with no Hindi, so the merge slider beyond that point changes nothing; real corpora are large enough to keep producing useful merges for a very long time, but with the same diminishing shape.

As an API user you do not choose the vocabulary, but you choose the model, and models differ in tokenizer. If a meaningful share of your traffic is in one language or script, include tokens per request for that traffic in the model comparison, not just the price per million tokens. A lower price per token can be a higher price per request.

A worked example: forecasting a multilingual assistant

Consider a hypothetical assistant for Belgavi's Book Store that handles one million requests a month with about 3,000 characters of input each (instructions, retrieved catalog snippets and the user's message, measured in English-equivalent content). Assume an illustrative input price of $2.50 per million tokens. Traffic is 60% English, 30% code-like content (structured catalog data and snippets), and 10% Hindi.

With the lab's English-first tokenizer (1,000 merges, no Hindi in training), English costs about 280 tokens per 1,000 characters, so an English request is about 840 tokens. Code-like content is about 1,000 tokens per request. A Hindi request carrying the same content costs 9.7 times the English tokens, about 8,200. The weighted average is 0.6 × 840 + 0.3 × 1,005 + 0.1 × 8,180 ≈ 1,620 tokens, or about $4,060 per million requests. The 10% of traffic in Hindi accounts for half the bill.

Add 20% Hindi to the training mix and fix the pre-tokenizer, and the Hindi request drops to roughly 1,180 tokens; the blended cost falls to about $2,300. If Hindi traffic then grows to 40%, the English-first tokenizer would cost about $9,500 per million requests while the multilingual one stays near $2,500. Every number here is calculated from the lab's synthetic corpus and price, but the method is what matters: measure tokens per request by segment with the production tokenizer, and forecast on the traffic mix you expect, not the one you launched with.

Knobs in real systems

If you call a hosted model, your knobs are measurement and model choice. Use the provider's own tokenizer or token-counting endpoint, not a different model's tokenizer and not a characters-divided-by-four rule; the ratio varies by language and content type. Log input and output token counts per request with language and route labels so you can see which segment drives cost.

If you train or adapt a tokenizer, the main controls look like this (check the exact names in the version you use):

# Hugging Face tokenizers: byte-level BPE
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
    vocab_size=64000,            # 256 bytes + merges + special tokens
    min_frequency=2,             # ignore pairs seen once
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
    special_tokens=["<|endoftext|>"])
tok.train(files=["en.txt", "code.txt", "hi.txt"], trainer=trainer)   # the mix is your policy

# SentencePiece: BPE with byte fallback
# spm_train --input=mix.txt --model_type=bpe --vocab_size=64000 \
#           --character_coverage=0.9995 --byte_fallback=true

The training mix is the most important decision and the easiest to get wrong by accident, because it is usually whatever text was convenient. Sample languages and content types deliberately, at least in proportion to the traffic you care about. Extending an existing model's vocabulary is possible but requires training new embeddings, so it is a model adaptation project, not a configuration change.

Failure modes and how to catch them

• Forecasts built on English. Cost and capacity models that assume four characters per token underestimate multilingual traffic. Detect it by tracking tokens per request by language.
• Context overflow on translation. A document that fits in English may not fit in another language. Size chunking and context budgets in tokens per language, not characters.
• Truncated outputs. A max_tokens limit sized for English answers cuts Hindi answers short. Watch the share of responses that stop on the length limit, by language.
• Silent regex splits. A pre-tokenizer that fragments a script makes that language expensive whatever the data. Inspect the tokens of a few real sentences per language; fragments of single characters are a warning sign.
• Whitespace bloat. Pretty-printed JSON and deeply indented code in prompts cost tokens that carry no meaning. Compare token counts before and after normalisation.
• Comparing prices per token across models. Different tokenizers make per-token prices incomparable. Compare cost per request on a sample of real traffic.

Most of these are cheap to catch: a weekly report of tokens per request and length-limit stops, broken down by language and route, finds them before the invoice does.

Run the lab: Tokenizer economics: train BPE merges and watch the bill change

The lab below generates a synthetic training corpus of English, Python-like code and Hindi, trains a real byte-level BPE tokenizer on it in your browser, and encodes held-out samples. The canvas replays the merges: each chip is one token, partial UTF-8 bytes appear as narrow dark chips, and you can watch words coalesce as more merges apply. The tiles turn token counts into tokens per 1,000 characters, a Hindi-to-English ratio for the same message, and an illustrative input cost per million requests. Start with the English-first default and look at the Hindi row.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. Count tokens for 50 real requests per language and per route with your production model's tokenizer, and record tokens per request.
2. Rebuild your cost forecast from tokens per request by segment and the traffic mix you expect in six months.
3. Check max_tokens and context budgets per language; alert on responses that stop at the length limit.
4. Normalise whitespace in structured payloads and retrieved code where semantics allow, and measure the saving.
5. When comparing models, compare cost per request on a sample of real multilingual traffic, not price per million tokens.
6. If you train a tokenizer, set the training mix deliberately and inspect the pre-tokenizer pattern on every script you serve.

Sources and further reading

• Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016)
• Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, byte-level BPE)
• Kudo and Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer (EMNLP 2018)
• Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
• Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (EMNLP 2023)
• Hugging Face Tokenizers documentation

Related in this series:

• Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
• Logprobs and Calibration: Measuring When a Language Model Is Guessing

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.