Why it matters

Whisper democratized high-quality ASR. Understanding capabilities enables voice-driven apps.

Advertisement

The architecture

Transformer encoder-decoder. Audio encoded to mel-spectrogram, encoded to features, decoded to text tokens.

Multilingual: 99 languages supported; auto-detect language.

Whisper architectureAudio → mel speclog-mel featuresEncoder-decodertransformerText outputwith language + timestampsModel sizes: tiny/base/small/medium/large-v3; larger = better + slower
Whisper components.
Advertisement

How it works end to end

Sizes: tiny (39M), base (74M), small (244M), medium (769M), large-v3 (1.5B). Trade quality vs speed.

Tasks: transcription (same language) and translation to English.

Timestamps: word-level or sentence-level timing.

Optimization: whisper.cpp (CPU), faster-whisper (CTranslate2), Whisper-JAX (TPU).