Why architecture matters here
Batch ASR is easy; streaming is hard. Batch has the whole utterance for context; streaming decides moment-to-moment. Poor endpointing cuts users off mid-sentence or holds them waiting. Unstable partials flicker letters in and out. High WER at low latency means agents misunderstand.
The architecture matters because tuning is joint: encoder chunk size affects latency AND accuracy; endpointing threshold affects both cut-off and hold time; rescore adds accuracy at the cost of some latency. Each pair trades off.
With the pipeline mapped, you can tune to your product's latency and accuracy SLOs.
The architecture: every piece explained
The top strip is the ingest and encode. Audio stream arrives at 16 kHz (typical). VAD filters silence and detects speech onset. Feature front-end produces mel-spectrograms or the equivalent for a conformer input. Streaming encoder uses chunked attention so it can emit outputs incrementally with limited lookahead.
The middle row is the decoder path. CTC / transducer decoders produce word / subword outputs monotonically over frames. Partial results are emitted continuously with confidence and word timings. Endpointing decides when the user has finished — silence duration plus intent signals like syntactic completeness. Rescore / LM fusion uses an external language model to correct partials and finalize the transcript with better lexical accuracy.
The lower rows are ecosystem pieces. Diarization + PII attaches speaker labels and scrubs sensitive content. Integration sends transcripts to chat, IVR, or caption UIs with the appropriate contract. Observability tracks word error rate (WER), latency (TTFT and end-of-word), endpoint precision, and partial churn (how much partials change before finalization).
End-to-end flow
End-to-end: a user says "what time is my flight tomorrow?" Audio streams in; VAD marks speech onset at 120 ms. Encoder emits chunked outputs; decoder produces partial "what", then "what time", then "what time is my flight tomorrow". Endpointing detects 400 ms of silence + syntactic completeness and closes the utterance. Rescore applies a small LM and the final transcript is confirmed. Total latency to final: 950 ms. Metrics: WER 4.2%, partial churn 6%, endpoint precision 0.97. Integration sends transcript to the agent runtime. If a caller trails off, endpointing waits longer than the base threshold because syntactic completeness is not reached.