Why it matters

Tokenization can bottleneck training and inference. Fast tokenizers keep it out of the critical path.

Advertisement

The architecture

Fast tokenizer: rust implementation. from_pretrained loads matching fast tokenizer if available.

Slow tokenizer: pure Python. Fallback when fast not available.

Tokenization algorithmsBPEGPT-style, byte-levelWordPieceBERT-styleUnigramSentencePieceAutoTokenizer picks the right one; fast=True selects Rust backend if available
Tokenization algorithms.
Advertisement

How it works end to end

Training tokenizers: build custom vocab for domain. Choose algorithm; feed corpus; save.

Special tokens: , , , , plus model-specific.

Encoding: batch encode returns tensors ready for model. Includes attention masks.

Offsets: fast tokenizers return character offsets for each token, enabling span extraction.