Why it matters

SentencePiece is popular multilingual tokenizer. Llama, T5 use.

Advertisement

The architecture

Raw text input.

BPE or Unigram.

Byte fallback for OOV.

SentencePiece deep diveRaw textno pre-tokenizeBPE or UnigramalgorithmByte fallbackOOV safetyKudo-Richardson 2018 Google; Llama + T5 + mT5 use; byte fallback safe
SentencePiece deep dive.
Advertisement

How it works end to end

Kudo-Richardson 2018.

Llama tokenizer.

T5, mT5.