Why it matters

WordPiece powered BERT. BPE dominant now; WordPiece for BERT compat.

Advertisement

The architecture

Likelihood-based merging.

## prefix for subwords.

Vocab typically 30k.

WordPiece tokenizationLikelihood merginggreedy selectSubword prefix## for continuationVocab 30kBERT tokenizerSchuster-Nakajima 2012 + Google BERT 2018; BPE more common today
WordPiece tokenization.
Advertisement

How it works end to end

Schuster-Nakajima 2012.

BERT 2018.

HuggingFace tokenizers.