Evidence label: deterministic synthetic model. The lab builds a synthetic corpus from word lists, a long tail of syllable-built rare words and code templates, in the proportions you choose. It splits the text with a GPT-2-style regular expression or one that keeps combining marks with letters, then trains byte-level BPE with incremental pair counts and applies the first m merges to held-out English, Hindi and code samples. The English and Hindi samples carry the same message. Token counts are real outputs of this small tokenizer; the corpus, traffic mix and price are synthetic and illustrative.
Play the lab
How to play
Move the merge slider to change vocabulary size, and the two mix sliders to decide how much code and Hindi the tokenizer is trained on (English is the remainder). Switch the pre-tokenizer between letters only and letters with combining marks. Each change retrains the tokenizer when the mix or pre-tokenizer changes, then re-encodes the samples; the canvas replays merges from zero to your setting. Inject the Hindi traffic growth or nested code events, try the presets, and use the missions: each starts from a broken tokenizer and asks for a measurable token or cost target, with stars for doing it with less training data or a smaller vocabulary.
What the model assumes
The training corpus has about 6,000 words. English and Hindi sentences mix common words, chosen with a Zipf-like skew, with rare words built from syllables; code comes from a few Python templates with four-space indentation. Some inflected words in the held-out samples are deliberately absent from the word lists. Requests are 3,000 English-equivalent characters; a Hindi request costs the English token count times the calculated ratio. Traffic is 60% English, 30% code and 10% Hindi, or 40/20/40 with the traffic event. The price is an illustrative $2.50 per million input tokens.
What it cannot tell you
A tokenizer trained on 6,000 synthetic words runs out of useful merges after a few hundred to a thousand and cannot reproduce the behaviour of production tokenizers trained on huge corpora, so absolute token counts and ratios will differ from any real model. The lab ignores normalisation, special tokens, unigram tokenizers and model quality effects. Use it to see the mechanism; use your provider's tokenizer on your own traffic to get numbers.
Read the full explanation
- Tokenizers Decide Your Bill: How BPE Merges Treat Languages and Code (companion article)
- Sampling Settings That Fix Real Failures: Temperature, Top-p and Min-p
- Logprobs and Calibration: Measuring When a Language Model Is Guessing
All AI in Production labs · Lab catalog
- Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016)
- Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, byte-level BPE)
- Kudo and Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer (EMNLP 2018)
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
- Ahia et al., Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (EMNLP 2023)
- Hugging Face Tokenizers documentation
Views are my own and do not represent any employer, past or present.