Adaptive jitter buffer architecture, in depth: the receive pipeline, decoder hand-off, mixing, lip sync and metrics

Three clocks, one place to absorb the difference

Three clocks meet in a receiver. The sender's capture clock is expressed in RTP timestamps, which count samples (48,000 per second for Opus). The network delivers packets at arrival times that carry the sender's spacing plus random queueing delay. The receiver's sound card consumes samples at its own crystal rate, through a callback that fires every 10 ms or so and must be fed immediately.

The buffer exists because these clocks disagree. Arrival time jitters around the send time, so the receiver waits a little before playing each packet. The sound card's crystal also differs from the sender's by tens of parts per million, so over a long call the receiver slowly gains or loses samples. Every other stage in the pipeline is designed to run at one of these clocks; the jitter buffer is the single place allowed to absorb the difference, by changing the length of the audio it outputs.

Three clocks, one place to absorb the difference

Three clocks meet in a receiver. The sender's capture clock is expressed in RTP timestamps, which count samples (48,000 per second for Opus). The network delivers packets at arrival times that carry the sender's spacing plus random queueing delay. The receiver's sound card consumes samples at its own crystal rate, through a callback that fires every 10 ms or so and must be fed immediately.

The buffer exists because these clocks disagree. Arrival time jitters around the send time, so the receiver waits a little before playing each packet. The sound card's crystal also differs from the sender's by tens of parts per million, so over a long call the receiver slowly gains or loses samples. Every other stage in the pipeline is designed to run at one of these clocks; the jitter buffer is the single place allowed to absorb the difference, by changing the length of the audio it outputs.

The receive pipeline and the pull contract

The receive pipeline has two halves with opposite rhythms. The network half is push-driven: a packet arrives, is decrypted (SRTP), demultiplexed by SSRC to the right stream, and inserted into that stream's packet store, ordered by sequence number. The device half is pull-driven: the audio device asks for the next 10 ms, the mixer asks each stream for 10 ms, and each stream's buffer must answer at once, either with decoded audio, stretched audio, or synthesised audio.

That pull contract is the most important rule in the design. The device callback runs on a real-time thread, so it must never block on a lock held by the network thread, never allocate, and never return short. Everything the buffer decides, it decides inside that call. In pseudocode:

def get_audio(stream, n_samples=480):          # 10 ms at 48 kHz, device thread
    while len(stream.out) < n_samples:
        pkt = stream.store.peek_next()           # lock-free queue from the network thread
        depth = stream.store.buffered_ms()
        target = stream.estimator.target_ms()
        if pkt is None or not pkt.due(stream.playout_ts):
            nxt = stream.store.peek_after_next()
            if nxt and nxt.has_fec:              # Opus in-band FEC: rebuild the missing frame
                stream.out += stream.dec.decode(nxt.payload, fec=True)
            elif stream.in_dtx:
                stream.out += stream.dec.comfort_noise()
            else:
                stream.out += stream.dec.conceal()  # PLC, counted as concealment
        elif depth > target + stream.hyst_ms:
            stream.out += stretch.accelerate(stream.dec.decode(pkt.payload))
        elif depth < target - stream.hyst_ms:
            stream.out += stretch.expand(stream.dec.decode(pkt.payload))
        else:
            stream.out += stream.dec.decode(pkt.payload)
        stream.playout_ts = stream.advance()
    return stream.out.take(n_samples)

The order of the branches carries the architecture. Recovering from redundancy comes before concealment, silence is handled before loss, and time-stretching only touches real decoded speech.

Handing off to the decoder: PLC, FEC and DTX

The buffer and the codec must agree on who handles each gap. Three codec features matter for Opus, the default voice codec in WebRTC; see packet loss concealment for how the synthesis itself works.

• Packet loss concealment. Calling the Opus decoder with no payload asks it to extrapolate the next frame from its internal state. It is good for one frame, tolerable for two or three, and turns robotic beyond that, so a sensible buffer fades to silence or comfort noise after about 60 to 100 ms of consecutive concealment.
• In-band FEC. When the encoder is told to expect loss, it embeds a lower-bitrate copy of the previous frame inside the current packet. If packet N is missing but N+1 has arrived, decoding N+1 in FEC mode recovers a real version of N. The buffer can only use this if N+1 is already present when N is due, which means it must hold at least one packet of look-ahead. A buffer tuned for minimum delay quietly throws this protection away.
• DTX. With discontinuous transmission the sender stops sending during silence, apart from occasional comfort-noise updates. Gaps in arrival are then expected, not loss, and the timestamp jumps when speech resumes. A buffer that treats DTX gaps as jitter or loss will inflate its target and report false concealment; it should instead know whether the stream is in DTX and generate comfort noise.

Redundant audio (RED, RFC 2198) is the explicit form of the same idea: each packet carries one or more older frames, and the buffer chooses the primary or the redundant copy, whichever arrives first. In every case the rule is the same: redundancy only helps if the buffer waits long enough to see it.

Many streams: per-stream buffers and the mixer

In a group call the receiver plays several streams at once, one per active speaker forwarded by an SFU. Each stream gets its own packet store, estimator and decoder state, because each crosses a different path from a different sender clock. A single shared buffer would force the worst sender's jitter onto everyone. The mixer then pulls 10 ms from each buffer, sums, and applies a limiter so that two loud talkers do not clip.

Per-stream buffers have costs. Decoding many streams costs CPU, so SFUs usually forward only the loudest few, chosen by the audio level header extension. A newly selected speaker arrives with an empty buffer and a cold estimator, so the first syllables can be clipped unless the buffer starts from a sensible prior rather than zero. The mixed output is also the far-end reference for the echo canceller: whatever the buffer stretches or conceals is what the canceller sees, so time-stretching happens before the reference tap, never after it.

Lip sync and the audio clock

Audio and video travel as separate RTP streams with unrelated timestamp bases. RTCP sender reports map each stream's RTP timestamps to a common wall clock, so the receiver can work out when a given audio sample and a given video frame were captured. If video needs more delay (because frames are decoded late or the video buffer is deeper), the receiver raises the audio buffer's minimum delay to match, and the reverse when audio is later.

Two consequences matter in practice. The audio buffer's observed delay is not purely a network measure, so comparing it across calls without knowing whether video was on is misleading. And audio is usually treated as the master clock: listeners notice audio glitches far more readily than a video frame held a few milliseconds longer, so the video side bends first.

Metrics and the one control

Browsers expose the buffer through getStats() on the inbound RTP stream. The delay counters are accumulated sums, so averages come from dividing by the count of emitted samples, and rates come from comparing two snapshots.

async function bufferReport(pc, prev = {}) {
  const out = {};
  (await pc.getStats()).forEach(s => {
    if (s.type !== "inbound-rtp" || s.kind !== "audio") return;
    const p = prev[s.id] || {};
    const d = k => (s[k] ?? 0) - (p[k] ?? 0);
    const emitted = d("jitterBufferEmittedCount");
    out[s.id] = {
      avgDelayMs: emitted ? 1000 * d("jitterBufferDelay") / emitted : null,
      targetMs: emitted ? 1000 * d("jitterBufferTargetDelay") / emitted : null,
      minimumMs: emitted ? 1000 * d("jitterBufferMinimumDelay") / emitted : null,
      concealedPct: 100 * d("concealedSamples") / Math.max(1, d("totalSamplesReceived")),
      stretchedSamples: d("insertedSamplesForDeceleration") + d("removedSamplesForAcceleration"),
      lateDiscards: d("packetsDiscarded"),
      raw: s,
    };
  });
  return out;
}

There is one control. RTCRtpReceiver.jitterBufferTarget lets the application state a preferred hold time in milliseconds, between 0 and 4000. It influences the buffer rather than setting it: the user agent still clamps the real target to what network conditions and memory allow. It is useful when an application knows something the estimator cannot, for example that a listen-only broadcast can afford 300 ms for smoothness. Browser support is not yet universal, so feature-detect it.

Worked example: diagnosing choppy calls

Suppose users on a support line report that agents sound choppy on some calls. Two snapshots ten seconds apart from an affected call give: average delay 38 ms, target 40 ms, minimum 36 ms, concealed 4.1 percent of samples, 0.3 percent silent concealment, 1,900 inserted and 2,100 removed samples, and 12 late discards against 3 losses.

Read it from the bottom. Late discards are four times the losses, so packets are arriving but after their deadline: this is jitter, not loss. The buffer is close to its target and its target is close to the minimum, so the estimator believes the network is calm. Combined with a high concealment rate, that points to bursty delay that a fast-decaying estimator forgets between bursts. Stretching in both directions within ten seconds confirms it is chasing the bursts. The remedies, in order of cost: slow the estimator's decay or raise its percentile; give the buffer enough look-ahead for in-band FEC and enable FEC on the sender; and, for an agent-side listen-heavy stream, set jitterBufferTarget to 80 ms and compare the concealment rate over a day of calls.

Failure modes

• Drift misread as jitter. A slow monotonic change in depth over minutes is clock drift; see clock drift compensation. Correct it gradually, not as a jitter event.
• DTX gaps counted as loss. Inflated targets and false concealment metrics during silence.
• Blocking in the callback. A lock shared with the network thread causes device underruns that look exactly like network loss in the audio but appear in no network statistic.
• Stretching after the echo reference tap. The canceller's reference no longer matches what was played, and residual echo appears only on bad networks.
• Cold start on speaker switch. First syllables clipped every time the SFU changes the forwarded set.
• Sequence wraps and sender restarts. A large jump in sequence number or timestamp must reset the stream, not register as thousands of lost packets.

Trade-offs

What to do next

1. Collect the getStats fields above every ten seconds from production calls and store the deltas.
2. Split concealment into silent and audible, and late discards from losses, before tuning anything.
3. Confirm that your sender enables in-band FEC and that your buffer keeps enough look-ahead to use it.
4. Check that DTX gaps do not move the target or count as concealment.
5. Audit the device callback for locks and allocation.
6. Verify that time-stretching happens before the echo canceller's reference tap.
7. Try jitterBufferTarget on one listen-heavy flow and compare concealment and delay for a week.