Why it matters

BPE is standard LLM tokenizer. tiktoken + SentencePiece implementations.

Advertisement

The architecture

Count byte pairs.

Merge most frequent.

Iterate to vocab size.

BPE tokenizationCount pairsin corpusMerge frequentiterativelyVocabtypically 32k-200kSennrich 2016; tiktoken + SentencePiece; foundation of GPT + Llama tokenizers
BPE tokenization.
Advertisement

How it works end to end

Sennrich 2016.

tiktoken OpenAI.

SentencePiece Google.