Why it matters

Unigram tokenizer probabilistic. SentencePiece implements; alternate to BPE.

Advertisement

The architecture

Initial large vocab.

Prune by likelihood.

Probabilistic segmentation.

Unigram LM tokenizerInitial vocablarge seedPrune by likelihooditerativeSegmentationprobabilisticKudo 2018; SentencePiece supports; probabilistic alternative to BPE
Unigram LM tokenizer.
Advertisement

How it works end to end

Kudo 2018.

SentencePiece.

Probabilistic sampling.