Why architecture matters here
VAD sits at the intersection of three budgets that pull in different directions. The latency budget: a voice agent that waits 1.5 seconds of silence before responding feels broken; endpointing pressure pushes offset detection earlier. The accuracy budget: cutting speech at the first quiet moment truncates utterances mid-sentence — the single most damaging error for ASR, since a clipped word is unrecoverable downstream. The compute/power budget: on always-listening devices, VAD is the component that runs 24/7, so it must cost milliwatts, which is why wake-word systems layer a near-zero-power energy detector under a small neural VAD under the (expensive) recognizer — each stage gating the next.
Architecture determines which trade-offs are even available. A per-frame classifier alone, however accurate, produces flickering output — speech probability dances around the threshold during fricatives and breaths — so the smoothing state machine is not a nicety but the difference between usable segments and confetti. Where you place VAD matters equally: client-side VAD saves uplink bandwidth and privacy (silence never leaves the device) but must survive device diversity — laptop fans, cheap MEMS microphones, echo from the device's own speaker; server-side VAD sees cleaner post-AEC audio and can afford bigger models, but everything streams regardless. Real systems usually run both: a permissive client gate and a precise server segmenter.
And in 2026's voice-agent stacks, the classic 'silence = end of turn' assumption is the top user complaint: humans pause mid-thought. That is why acoustic VAD is being augmented with semantic turn detection — a model that reads the partial transcript (and prosody) to judge whether the utterance is complete, not merely whether sound stopped. Knowing where pure VAD ends and turn policy begins is now a core architecture decision.
The architecture: every piece explained
Top row: from air to score. Capture standardizes to mono 16kHz PCM (8kHz for telephony), ideally after acoustic echo cancellation and noise suppression in the device's audio front-end — VAD accuracy is inherited from this chain more than from the model. Framing slices the stream into 10–30ms windows, typically with overlap; frame size sets the resolution/latency floor. The feature stage is model-dependent: classic VADs compute energy, zero-crossing rate, and spectral features; WebRTC's GMM VAD uses sub-band energies; modern neural VADs consume log-mel spectrograms or raw waveforms directly. The model ladder trades accuracy for cost: energy thresholds (free, fooled by any noise), WebRTC GMM (tiny, aggressive-mode still noise-fragile), and small DNNs — Silero VAD's few-MB network is the de-facto open standard — which stay robust in cafes, cars, and crosstalk at ~1ms per frame on a CPU core.
Middle row: from scores to segments. The model emits a per-frame speech probability. Hysteresis thresholds convert it to states: enter speech when probability exceeds the onset threshold (say 0.6), exit only when it falls below a lower offset threshold (say 0.35) — the gap prevents flicker at the boundary. The smoothing state machine adds time rules: min speech duration (drop 'speech' bursts shorter than ~100ms — door clicks, keyboard taps), min silence duration / hangover (stay in speech for 200–500ms after scores drop, bridging stop-consonant gaps and breath pauses), and pre-roll padding (emit 100–300ms of audio from before the detected onset, recovering the soft first syllable the detector needed a few frames to notice — the single cheapest accuracy win in the whole pipeline). Output is clean segment events: speech-start with buffered pre-roll, speech-end with timestamps.
Bottom rows: consumers and the modern extension. ASR receives only speech (cost and accuracy both improve — recognizers hallucinate on long silence); endpointing turns speech-end into 'finalize the utterance'; diarization uses segments as its unit of clustering; codecs use VAD for discontinuous transmission (send comfort noise instead of encoded silence). Turn detection layers policy on top: a semantic model scores transcript completeness so a 700ms mid-sentence pause doesn't trigger the agent, while a completed question triggers it after 300ms — dynamic endpointing in place of one fixed silence timeout. The ops strip is the honest footer: every deployment environment (call center headsets, car cabins, open-plan offices) needs its own threshold and hangover calibration, validated against labeled audio from that environment.
End-to-end flow
Trace a real-time voice agent session. The user's browser captures 16kHz audio; the WebRTC stack applies echo cancellation (critical — the agent's own voice is playing from the same device) and a client-side Silero VAD gates the uplink: silence frames are dropped, so the connection carries near-zero bytes while nobody speaks.
The user starts: 'What's the… um… status of my order?' Frame scores rise past 0.6 within three frames of the first vowel; the state machine flips to SPEECH and emits speech-start with 200ms of pre-roll from its ring buffer — the breathy 'Wh' that scored only 0.4 is included. Audio streams to the server, where streaming ASR begins emitting partial transcripts. At the 'um…' the user pauses 600ms: frame scores drop to 0.1, but the hangover window (350ms) expires and acoustic VAD reports a tentative speech-end. Here the turn detector earns its keep: the partial transcript 'What's the' is scored incomplete (a determiner with no noun phrase), so the endpointer extends its patience window rather than finalizing — the agent stays quiet. Speech resumes; the segments bridge into one utterance.
The user finishes: '…status of my order?' Scores drop; after 300ms of silence the semantic model reads a complete question with terminal rising pitch and endorses the endpoint immediately — no need to wait the full default timeout. Speech-end fires; ASR finalizes; the LLM responds; TTS plays. Mid-response the user interrupts — 'actually, cancel it' — and the client VAD's speech-start doubles as the barge-in signal: TTS playback is killed within ~100ms (possible only because echo cancellation prevents the agent's own voice from triggering its VAD), and the turn flips back to the user.
The same segments do double duty downstream: they are logged with timestamps, so diarization in the analytics pipeline clusters them by speaker embedding, per-call speech/silence ratios feed QA dashboards, and ASR is billed for 41 seconds of speech out of a 3-minute call. Every behavior above — the pre-roll saving 'Wh', the 'um' surviving the pause, the fast endpoint on a complete question, the barge-in — is a specific architectural choice from the previous section doing its one job.