TOKENIZERS · BYTES PER TOKEN
Tokenizer Compression Comparison
Train a byte-level BPE tokenizer on a short English text right here, encode your own text with it, and see why other languages and code cost more tokens.
Your experiment
Start with the English sample and 100 merges. Read the tokens, then switch to code, German, Hindi, numbers and emoji, set the merges to 0 and 150, turn off pre-splitting, and paste text of your own.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Tokens and the first merges
Computed data
Metrics
The tokenizer is trained in your browser on a built-in English paragraph of about 1,300 bytes: 256 byte tokens plus merges of the most frequent adjacent pair, ties broken by the lower pair id, stopping when no pair occurs twice. Real tokenizers learn 32,000 to 256,000 tokens from billions of words, so their bytes per token are higher, but the pattern is the same: text unlike the training data falls back towards one token per byte. Use your model's own tokenizer to count real costs.
A tokenizer trained in the page
Byte-level BPE starts with 256 tokens, one per byte, so any text can be encoded. Training repeatedly finds the most frequent pair of adjacent tokens in the training text and merges it into a new token. Here the training text is a short English paragraph, and the first merges are a space with t, h with e, then the space-t piece with he: the building blocks of common English words. Encoding applies the learned merges in the order they were learned. Choosing a Sample text, or typing into Text to encode, changes only what is encoded: the tokenizer is always trained on the same built-in English paragraph.
English compresses well
Merges to learn sets how many merges training may add on top of the 256 byte tokens. With 100 merges the English sample, 114 bytes, becomes 38 tokens, 3 bytes per token, and common words such as the, tokens and training are single tokens. With 0 merges every byte is a token, 114 tokens: BARELY COMPRESSES. At 150 the tokenizer stops at 148 merges, because after that no pair occurs twice in the training text, and the sample needs 32 tokens, 3.56 bytes each.
Other text costs more
The same 100 merges give Python code 1.48 bytes per token, German 1.44 and numbers 1.21: these pieces were rarely or never seen in training. Hindi is the extreme case. Its 49 characters take 133 UTF-8 bytes, 3 bytes for most Devanagari characters, and none of their byte pairs were learned, so it costs 133 tokens, 2.71 tokens per character. The emoji sample, 27 characters and 36 bytes, needs 29 tokens. Context limits and prices are counted in tokens, so the same message costs more in an under-represented script.
Pre-splitting
Pre-split into words before merging cuts both the training text and the text to encode into words, numbers and punctuation before merging, so a merge never crosses a word boundary; GPT-style tokenizers do this with a regular expression. Turn it off and merges can join the end of one word to the start of the next. With 100 merges that encodes the English sample in 40 tokens instead of 38, because some merges are spent on cross-word pieces that do not recur in a new sentence.
Reading the tool and its limits
The bars compare bytes per token for every sample against 2, the line below which the result is called BARELY COMPRESSES; the lanes summarise your text; the table lists every sample; and the text panel shows the tokens, with spaces as · and unlearned bytes in hex, and the first 12 merges. A real tokenizer trained on far more text compresses better, so use these numbers for the pattern, not for cost estimates.