Real-Time Audio Architecture in Depth

The audio callback contract

Every audio API, whether Core Audio on Apple platforms, WASAPI on Windows, AAudio on Android or ALSA and PipeWire on Linux, eventually calls your code on a high-priority thread with a buffer to fill or drain. The buffer size is a number of frames, and its duration is frames divided by sample rate: 128 frames at 48 kHz is 2.67 ms, 480 frames is 10 ms. That duration is your whole budget, and you share it with the operating system.

The rule that follows is simple and strict: the callback must have bounded execution time. That rules out anything whose worst case is unbounded, which is more than it sounds like.

• No memory allocation or freeing; allocators take locks and occasionally call into the kernel.
• No mutexes. If a lower-priority thread holds the lock and is preempted, the audio thread waits on it. This is priority inversion, and it produces glitches that only appear under load.
• No file or network I/O, no logging, no system calls that can block, and no waiting for page faults: touch and pre-fault buffers before starting the stream.
• No garbage-collected or interpreted code on this thread unless the runtime guarantees no pauses.

The standard way to honour the rule is a single-producer single-consumer ring buffer between the callback and a normal thread. Each side owns one index and only reads the other, so atomic loads and stores with acquire/release ordering are enough; no compare-and-swap loops and no locks are needed.

#include <atomic>
#include <cstddef>
#include <vector>

class SpscRing {
public:
    explicit SpscRing(size_t capacity) : buf_(capacity + 1) {}   // one slot kept empty

    // Producer side only (e.g. mic callback).
    size_t write(const float* src, size_t n) {
        size_t w = write_.load(std::memory_order_relaxed);
        size_t r = read_.load(std::memory_order_acquire);
        size_t free = (r + buf_.size() - w - 1) % buf_.size();
        if (n > free) n = free;                       // drop rather than block
        for (size_t i = 0; i < n; ++i) buf_[(w + i) % buf_.size()] = src[i];
        write_.store((w + n) % buf_.size(), std::memory_order_release);
        return n;
    }

    // Consumer side only (e.g. processing thread).
    size_t read(float* dst, size_t n) {
        size_t r = read_.load(std::memory_order_relaxed);
        size_t w = write_.load(std::memory_order_acquire);
        size_t avail = (w + buf_.size() - r) % buf_.size();
        if (n > avail) n = avail;
        for (size_t i = 0; i < n; ++i) dst[i] = buf_[(r + i) % buf_.size()];
        read_.store((r + n) % buf_.size(), std::memory_order_release);
        return n;
    }

private:
    std::vector<float> buf_;                          // allocated once, before the stream starts
    std::atomic<size_t> write_{0}, read_{0};
};

Note what happens when the ring is full or empty: the callback drops or pads, and increments a counter that another thread reports later. A real-time thread degrades; it never waits. On Windows, register the processing thread with MMCSS (the "Pro Audio" task) and on Linux request real-time scheduling through rtkit, so that the soft-deadline stages also get predictable CPU time.

The audio callback contract

Every audio API, whether Core Audio on Apple platforms, WASAPI on Windows, AAudio on Android or ALSA and PipeWire on Linux, eventually calls your code on a high-priority thread with a buffer to fill or drain. The buffer size is a number of frames, and its duration is frames divided by sample rate: 128 frames at 48 kHz is 2.67 ms, 480 frames is 10 ms. That duration is your whole budget, and you share it with the operating system.

The rule that follows is simple and strict: the callback must have bounded execution time. That rules out anything whose worst case is unbounded, which is more than it sounds like.

• No memory allocation or freeing; allocators take locks and occasionally call into the kernel.
• No mutexes. If a lower-priority thread holds the lock and is preempted, the audio thread waits on it. This is priority inversion, and it produces glitches that only appear under load.
• No file or network I/O, no logging, no system calls that can block, and no waiting for page faults: touch and pre-fault buffers before starting the stream.
• No garbage-collected or interpreted code on this thread unless the runtime guarantees no pauses.

The standard way to honour the rule is a single-producer single-consumer ring buffer between the callback and a normal thread. Each side owns one index and only reads the other, so atomic loads and stores with acquire/release ordering are enough; no compare-and-swap loops and no locks are needed.

#include <atomic>
#include <cstddef>
#include <vector>

class SpscRing {
public:
    explicit SpscRing(size_t capacity) : buf_(capacity + 1) {}   // one slot kept empty

    // Producer side only (e.g. mic callback).
    size_t write(const float* src, size_t n) {
        size_t w = write_.load(std::memory_order_relaxed);
        size_t r = read_.load(std::memory_order_acquire);
        size_t free = (r + buf_.size() - w - 1) % buf_.size();
        if (n > free) n = free;                       // drop rather than block
        for (size_t i = 0; i < n; ++i) buf_[(w + i) % buf_.size()] = src[i];
        write_.store((w + n) % buf_.size(), std::memory_order_release);
        return n;
    }

    // Consumer side only (e.g. processing thread).
    size_t read(float* dst, size_t n) {
        size_t r = read_.load(std::memory_order_relaxed);
        size_t w = write_.load(std::memory_order_acquire);
        size_t avail = (w + buf_.size() - r) % buf_.size();
        if (n > avail) n = avail;
        for (size_t i = 0; i < n; ++i) dst[i] = buf_[(r + i) % buf_.size()];
        read_.store((r + n) % buf_.size(), std::memory_order_release);
        return n;
    }

private:
    std::vector<float> buf_;                          // allocated once, before the stream starts
    std::atomic<size_t> write_{0}, read_{0};
};

Note what happens when the ring is full or empty: the callback drops or pads, and increments a counter that another thread reports later. A real-time thread degrades; it never waits. On Windows, register the processing thread with MMCSS (the "Pro Audio" task) and on Linux request real-time scheduling through rtkit, so that the soft-deadline stages also get predictable CPU time.

The speech processing chain

Between the capture ring and the encoder sits the speech processing chain. The WebRTC audio processing module, the most widely deployed open implementation, works on 10 ms frames, and the order of its stages matters.

1. Acoustic echo cancellation (AEC) subtracts an estimate of the far-end signal that the speaker played and the microphone picked up again. It needs that far-end signal as a reference, taken from the render path as close to the speaker as possible, plus a stable estimate of the delay between playing and capturing it. If the reference is missing or the delay estimate wanders, the other party hears themselves.
2. Noise suppression (NS) attenuates stationary and, with neural suppressors, non-stationary noise. It runs after AEC so it does not have to model echo as noise.
3. Automatic gain control (AGC) normalises level so quiet talkers are audible and loud ones do not clip the encoder.
4. Voice activity detection (VAD) labels frames as speech or not. The codec can use it for discontinuous transmission, and voice agents use it to decide when a user has started or stopped talking.

These stages run on the processing thread, not in the callback, because their cost varies with the signal, and a neural suppressor can take several milliseconds on a slow core. The ring buffer absorbs that variation.

Codec framing and bitrate

Opus, standardised in RFC 6716, is the default codec for interactive voice on the web and in most VoIP stacks. It supports frame sizes from 2.5 to 60 ms; 20 ms is the common choice because it balances packet overhead against delay. Each packet carries roughly 40 bytes of IP, UDP and RTP headers, so at 20 ms frames the headers alone are about 16 kbps of overhead, and at 10 ms frames about 32 kbps. Voice at 24 to 32 kbps of payload is transparent for most listeners; wideband speech remains intelligible at much lower rates.

Packetization delay is the frame duration plus the codec's look-ahead, so the frame size you choose is paid once at the sender. Opus lets the encoder adapt bitrate on the fly, and in WebRTC the congestion controller (transport-wide congestion control feedback driving a bandwidth estimator) tells the encoder how much it may send. The encoder must also be told the expected loss rate, because that controls how much redundancy it embeds.

Recovering from packet loss

Packet loss is handled by three distinct mechanisms that are often confused.

Parity-based schemes such as ULPFEC and FlexFEC exist too, but are mostly used for video. The receive loop for Opus shows how FEC and PLC fit together. It uses the real libopus call opus_decode(dec, data, len, pcm, frame_size, decode_fec).

// Called once per 20 ms playout tick, pulling from the jitter buffer.
int next_frame(OpusDecoder* dec, JitterBuffer* jb, uint32_t seq, opus_int16* pcm, int frame) {
    Packet* cur = jb_get(jb, seq);
    if (cur)                                   // normal case
        return opus_decode(dec, cur->data, cur->len, pcm, frame, 0);

    Packet* nxt = jb_get(jb, seq + 1);         // lost: does the next packet carry FEC for it?
    if (nxt)
        return opus_decode(dec, nxt->data, nxt->len, pcm, frame, 1);

    return opus_decode(dec, NULL, 0, pcm, frame, 0);   // nothing to recover from: conceal
}

On the sender side, in-band FEC is switched on with OPUS_SET_INBAND_FEC(1) and tuned with OPUS_SET_PACKET_LOSS_PERC(n); with a loss estimate of zero the encoder may decide not to spend bits on redundancy at all. Feed it the loss rate reported by the receiver.

Jitter buffers and clock drift

Packets leave the sender every 20 ms but arrive irregularly. The jitter buffer holds them long enough to reorder and smooth arrival times, then releases one frame per playout tick. Its target depth is a direct trade: deeper means fewer late packets treated as lost, shallower means less delay. Adaptive buffers track the recent distribution of arrival delay and aim for a high percentile of it, growing quickly when jitter rises and shrinking slowly afterwards. They change depth by time-stretching speech, shortening or lengthening pauses, rather than by dropping or inserting samples, which would click. The design is covered in depth in the jitter buffer article.

Drift is the quieter problem. The sender's microphone clock and the receiver's speaker clock are separate crystals that disagree by tens of parts per million. At 100 ppm and 48 kHz that is 4.8 samples per second, or about 360 ms per hour, so without correction a long call either starves or overflows the buffer. The same time-stretching or an asynchronous resampler absorbs the difference; clock drift compensation explains the estimators.

Voice agents and barge-in

Voice agents add a consumer and a constraint. The consumer is speech recognition, which reads near-end PCM after AEC and NS. The constraint is barge-in: when the user starts speaking while synthesised speech is playing, the system should stop talking within a couple of hundred milliseconds. That requires three things to happen together: VAD fires on the near-end signal, the text-to-speech and language model requests are cancelled, and the playout ring and jitter buffer are flushed so queued audio does not keep playing.

Barge-in only works if echo cancellation is good, because otherwise the agent hears its own voice through the microphone and interrupts itself. Feed the TTS output into the AEC reference exactly as you would a remote participant, and test with real laptop speakers rather than headphones.

Worked example: a one-way latency budget

Take a voice assistant on a laptop talking to a server over a good broadband link and add up one direction of the budget.

The total lands near the 150 ms guideline before the server has done any work, which is the key insight: for a voice agent, recognition, reasoning and synthesis must stream, emitting partial results and first audio as early as possible, because the transport alone uses most of the conversational budget.

Observability

Measure glitches and delay directly rather than inferring them from CPU graphs. In the browser, WebRTC getStats() inbound audio statistics include concealedSamples, totalSamplesReceived, jitterBufferDelay and jitterBufferEmittedCount; the ratio of the first two is your concealment rate, and the ratio of the last two is average jitter buffer delay. In native code, count underruns and overruns in the ring buffer, and record a histogram of callback execution time against the period.

Failure modes

• Clicks under load. The callback blocked on a lock, an allocation or a page fault. Audit the callback, pre-fault buffers, and count xruns.
• Echo after a device change. Switching to Bluetooth changes output latency by 100 ms or more, and the AEC delay estimate is stale. Reset or re-converge AEC on device changes.
• Slowly growing delay. Uncorrected clock drift or a jitter buffer that grows and never shrinks. Plot jitter buffer delay over the length of a call.
• Robotic audio on mobile networks. Long bursts defeat in-band FEC and stretch PLC. Consider RED for voice-critical calls and confirm the encoder receives the measured loss rate.
• Agent interrupts itself. TTS output is not in the AEC reference path.

What to do next

1. Write down your one-way latency budget per stage and measure each stage, not just the total.
2. Audit every audio callback for allocation, locks, logging and blocking calls; move work behind an SPSC ring.
3. Pick a device buffer and Opus frame size together, and record the resulting framing delay.
4. Enable Opus in-band FEC and wire the receiver's loss estimate into the encoder; add RED where bursts are common.
5. Export concealment rate, jitter buffer delay and xrun counts, and alert on them per call.
6. For voice agents, route TTS through the AEC reference and test barge-in on real speakers.
7. Read on: the Opus codec, FEC and redundancy, acoustic echo cancellation and buffer management for low latency.