Why architecture matters here
ASR architecture matters because speed and accuracy trade off. Batch Whisper is accurate but slow; streaming ASR is fast but lower accuracy. Real-world apps often need both — real-time captions during a call, then a polished transcript after.
Cost is per audio hour. Cloud APIs charge $0.30-$2 per hour; self-hosted Whisper on GPU is cheaper for high volume.
Reliability is where careful design shows. Silence handling, VAD chunking, and streaming boundaries all affect user perception.
The architecture: every stage explained
Walk the diagram top to bottom.
Audio Input. Usually 16kHz mono PCM. Higher sample rates typically resampled.
Feature Extraction. Log-mel spectrogram: STFT then mel-filterbank + log. Compresses audio to a matrix suitable for neural network input.
Encoder. Conformer or Transformer that turns log-mel features into contextual embeddings. Handles time dimension.
Decoder. Autoregressive; produces text tokens from encoder output + previous tokens. Whisper uses BPE-tokenized text.
Language Model biasing. Bias toward likely tokens for domain (medical, legal, custom vocab).
Streaming Decoding. Process audio in chunks; emit partial hypotheses; refine. Faster-Whisper + custom kernels.
Whisper family. Whisper (large, medium, small, tiny), Distil-Whisper (small, fast), Whisper-turbo (efficient inference), Whisper v3.
Diarization. Who spoke when — separate speakers. pyannote-audio, WhisperX are standard.
Voice Activity Detection. Skip silence; segment into utterances. Silero VAD common.
Post-processing. Add punctuation, true-case, numbers ("twenty twenty-six" → "2026").
End-to-end real-time flow
Trace a real-time ASR. Live meeting; VAD detects speech starting. Audio streamed at 16kHz.
Every 500ms, latest 30s window sent to streaming Whisper. Feature extraction produces log-mel.
Encoder processes; decoder generates partial transcript. Emit to UI as caption. Refine on next chunk (delayed context improves accuracy).
Diarization runs in parallel on same audio; identifies speaker changes. Combined output: "Speaker 1: Hello everyone."
Meeting ends. Full audio sent to non-streaming Whisper for polished transcript. Better accuracy since full context available. Post-processing adds punctuation + numbers + speaker labels.
Cost breakdown: streaming (accurate enough for captions) cheaper than large model per chunk; full-pass polished after cheaper because batched.
Language: Whisper multilingual. User speaks in German; model detects and transcribes. English speaker joins; auto-detect switches.