Why architecture matters here
Real-time audio architecture matters because human ears are unforgiving. Mouth-to-ear latency above 200 ms breaks conversational flow. Packet loss above 1% degrades intelligibility. Echo makes calls unusable. Each of these has a specific remedy in the pipeline; miss one and users notice.
Cost matters because voice at scale is bandwidth-intensive. A well-tuned Opus stream is 24-32 kbps; a poorly configured one is 128 kbps for the same quality. Multiplied by concurrent calls, the difference is measurable.
Reliability under network variance is where great pipelines separate from average ones. Adaptive jitter buffers, forward error correction, and packet loss concealment all soften the impact of imperfect networks. Users on mobile, coffee shop WiFi, and hotel networks depend on this work.
The architecture: every stage explained
Walk the diagram top to bottom.
Mic Capture. Raw PCM audio at 16 kHz (voice-quality) or 48 kHz (music-quality). Sample rate choice trades bandwidth vs fidelity. Voice-only apps standardize on 16 kHz; conferencing on 48 kHz.
Preprocessing. Acoustic Echo Cancellation (AEC) removes the far-end audio that leaks back into the mic. Noise Suppression (NS) removes background noise. Voice Activity Detection (VAD) identifies speech vs silence, letting the codec skip silence packets and letting the app know when someone is talking.
Codec. Opus is the default for real-time speech and music; AAC for streaming; G.711 for telephony compatibility. Opus dynamically adjusts bitrate based on network feedback (via RTCP).
Packetizer + FEC. Encoded audio is packaged into 20 ms RTP packets (default for Opus). Forward Error Correction adds redundancy — receiving N packets of every M lets you reconstruct despite loss. FEC costs bandwidth but saves the call over lossy networks.
Transport. WebRTC uses SRTP over UDP with congestion control (GCC or Transport-CC). SRT and QUIC are alternatives with different trade-offs. Congestion control is what keeps the stream adapting to available bandwidth.
Network. Packet loss (0.1-5% typical), jitter (0-50 ms), and delay (10-300 ms) vary continuously. The stream must survive.
Jitter Buffer. Receiver-side. Accumulates packets briefly to smooth out arrival variance. Adaptive buffers grow under high jitter and shrink under low jitter to minimize latency. Packet Loss Concealment (PLC) invents plausible audio for missing packets.
Decoder. Converts encoded frames back to PCM. On loss, PLC produces synthesized audio (usually inaudible for short gaps).
Speaker Output / ASR. Two consumers of decoded audio. Speaker output for playback; ASR for speech-to-text (voice assistants, agents). Both may consume the same stream in parallel.
Barge-in Detection. For voice agents: detect when the user starts speaking while the assistant is talking. Cut off the assistant, cancel any in-flight TTS, and switch to listening. Barge-in makes voice agents feel natural.
Metrics. Mean Opinion Score (MOS) subjective quality proxy. Mouth-to-ear latency end-to-end. Packet loss and jitter continuously. Alert when quality drops below thresholds.
End-to-end call flow
Trace a call. User A speaks. The mic captures at 48 kHz. AEC removes the audio A's speaker is playing (from user B). Noise suppression cleans it up. VAD flags speech-active.
The Opus encoder produces 20 ms frames at 32 kbps. FEC adds redundancy. Frames become RTP packets, encrypted as SRTP, sent over UDP to the transport partner (peer, SFU, or MCU).
Packets traverse the network. Some arrive late; some are lost. User B's receiver ingests them, decrypts, and pushes to the jitter buffer. The buffer holds packets for 60 ms to absorb jitter, dynamically adjusting.
The decoder pulls frames from the buffer. On loss, PLC synthesizes a bridge. Frames become PCM again and go to the speaker.
Simultaneously the decoded PCM feeds an ASR service. Whisper or a streaming ASR produces transcripts. The transcripts drive UI captions or feed the LLM's next response.
When the LLM starts generating a response and TTS is playing back to user A, user A speaks again. Barge-in detection fires: the TTS stops mid-word, the LLM stops generating, listening resumes.
End-to-end mouth-to-ear latency: 150-200 ms typical. MOS: 4.2 on well-tuned pipelines. Users experience natural conversation.