Why architecture matters here
A fixed jitter buffer is a single number pretending to be a policy. Set it to 40ms and users on clean fiber pay 40ms of avoidable mouth-to-ear latency on every call, while users on congested Wi-Fi — where delay spikes routinely exceed 100ms — hear constant underruns anyway. There is no single value that serves both, and the same user's network changes minute to minute as they walk between access points or a neighbor starts a download. The buffer must therefore be a feedback controller, not a constant.
The stakes are asymmetric and perceptual. ITU-T G.114 puts the conversational comfort threshold around 150ms one-way; every millisecond of buffer depth spends directly against that budget, on top of capture, encode, network, decode, and render delays that already consume most of it. Underruns, meanwhile, force the system to invent audio — concealment sounds fine for one lost 20ms frame and increasingly synthetic beyond three. Quality is maximized not by minimizing either latency or loss alone but by tracking the knee of the tradeoff as the network moves it.
Architecture matters here because the buffer sits at the junction of three clock domains: the sender's capture clock, the network's arrival process, and the receiver's playback clock. None agree. Sequence numbers and RTP timestamps let the buffer reconstruct sender intent; arrival-time statistics characterize the network; and the playout side must honor the sound card's inflexible appetite for samples. A design that muddles these domains — for example, resizing the buffer by dropping packets rather than time-stretching — produces artifacts that no amount of tuning removes. The clean separation of estimation, scheduling, and signal manipulation is the whole game.
The architecture: every piece explained
RTP receiver and reorder buffer. Packets arrive carrying a sequence number (detects loss and reordering) and a media timestamp (position in the audio stream, in sample ticks). The reorder buffer inserts each packet by sequence number, so a packet that arrives late but before its playout deadline is simply slotted into place — reordering is free until the deadline passes. Duplicates are discarded; a large forward jump in sequence numbers is flagged as a probable sender restart rather than a billion lost packets.
Delay estimator. For each arrival the buffer computes inter-arrival jitter: the difference between how far apart two packets arrived and how far apart their timestamps say they were sent. These deltas feed a histogram — NetEQ uses an exponentially-decaying histogram of delay in packet units — and the target buffer depth is chosen as a high percentile, typically the 95th to 99th. The percentile choice is the latency/loss dial: P95 accepts that 5% of packets will arrive too late and be concealed, in exchange for a shallower buffer. Decay rate sets memory: fast decay adapts quickly but overreacts to single spikes.
Playout scheduler. Every audio-device callback (10 or 20ms), the scheduler compares current buffer depth to target and picks an operation: normal decode when on target; accelerate (time-compress speech, removing pitch periods) when the buffer is too deep; preemptive expand (stretch) when it is running dry; concealment when the next packet is missing entirely. Time-stretching via WSOLA-style overlap-add changes duration without changing pitch, so adaptation is inaudible at moderate rates.
Clock-drift compensator and controller. Sender and receiver sound cards drift — 50ppm apart means the buffer gains or loses ~180ms per hour. The drift estimator detects a slow monotonic trend in buffer depth, distinct from jitter, and bleeds it off through the same accelerate/expand machinery. The adaptation controller wraps everything with mode logic: spike mode when delay jumps by hundreds of milliseconds, DTX awareness so comfort-noise silence is not mistaken for loss, and hard min/max clamps from the application.
End-to-end flow
Follow one packet through a call. The sender captures 20ms of speech, encodes it with Opus, stamps RTP sequence 17,204 and timestamp 344,080 (sample ticks at 16kHz), and sends. The network adds 38ms this time — 12ms more than the previous packet. On arrival, the receiver records the arrival delta versus the timestamp delta, updates the jitter histogram (this 12ms excursion nudges the tail), and inserts the packet into the reorder buffer by sequence number. The estimator recomputes: the 97th percentile of recent jitter now says the target depth is three packets, 60ms.
Meanwhile the sound card fires its 10ms callback. The scheduler checks depth: 80ms buffered against a 60ms target — slightly deep, but within hysteresis, so it decodes normally and hands samples to the mixer. Two ticks later a Wi-Fi retransmission burst delays everything; depth falls to 20ms. The scheduler switches to preemptive expand, stretching the last decoded frame by 30% — the buffer refills during the stretched playout without a single dropped or invented frame. When the burst clears and six packets land at once, depth spikes to 140ms; accelerate mode removes pitch periods over the next second, walking depth back to 60ms so quickly the listener never notices the call was briefly a fifth of a second behind.
Now packet 17,207 never arrives. Its playout deadline passes; the scheduler invokes concealment, which extrapolates the previous frame's spectral envelope and pitch. One concealed frame is inaudible. When 17,208 arrives normally, the decoder cross-fades from concealed audio back to real audio. If 17,207 then shows up 90ms late, it is simply discarded — its moment has passed. Every path through the machine ends with the sound card getting exactly the samples it demanded, on time, every tick.