Why it matters
Token count determines cost (per-token pricing) and context usage. Understanding tokenization helps optimize prompts and diagnose issues.
The architecture
BPE tokenization: start with characters. Iteratively merge most-common pairs into new tokens. Final vocab has 50K-200K tokens: common words as single tokens, rare words as multiple.
Encode: split text into token sequence via vocab lookup.
How it works end to end
English typically 1 token per 3-4 chars. Non-English worse: Chinese, Japanese sometimes 1 token per char.
Code tokenizes differently: symbols and keywords often single tokens, whitespace matters.
Numbers tokenize inconsistently: '2024' might be one token; '01/15/2024' might be many.
Different models use different tokenizers (BPE variants, SentencePiece). Same text → different token counts.