Acoustic echo cancellation architecture, in depth: reference, delay, adaptive filter, double talk and residual suppression

Why echo is an architecture problem

Start from the signal model. The microphone signal is the sum of three things: near-end speech s(n), noise v(n), and echo. The echo is the far-end signal x(n) after it has passed through the loudspeaker, the device body and the room. To a good approximation that path is a linear filter with impulse response h, so the echo is h * x and the microphone sees d(n) = (h * x)(n) + s(n) + v(n).

Because the canceller has x (it is the audio the device itself is playing), it can learn an estimate of h, predict the echo, and subtract it. The error e(n) = d(n) - y(n) is both the output sent onward and the signal used to adapt the filter. When the estimate is good, e is roughly s + v: the user's voice survives and the echo does not.

Three facts turn that simple idea into an architecture. First, h is long: a room's echo tail commonly runs to a few hundred milliseconds, thousands of taps at speech sample rates. Second, h changes whenever someone moves, the volume changes or a door opens, so the filter must keep tracking. Third, whenever the near-end user talks, s is in the error signal, and a filter that adapts on it will start cancelling the user. Every component below exists to deal with one of those three facts, under a budget of a few milliseconds of compute per 10 ms frame.

Why echo is an architecture problem

Start from the signal model. The microphone signal is the sum of three things: near-end speech s(n), noise v(n), and echo. The echo is the far-end signal x(n) after it has passed through the loudspeaker, the device body and the room. To a good approximation that path is a linear filter with impulse response h, so the echo is h * x and the microphone sees d(n) = (h * x)(n) + s(n) + v(n).

Because the canceller has x (it is the audio the device itself is playing), it can learn an estimate of h, predict the echo, and subtract it. The error e(n) = d(n) - y(n) is both the output sent onward and the signal used to adapt the filter. When the estimate is good, e is roughly s + v: the user's voice survives and the echo does not.

Three facts turn that simple idea into an architecture. First, h is long: a room's echo tail commonly runs to a few hundred milliseconds, thousands of taps at speech sample rates. Second, h changes whenever someone moves, the volume changes or a door opens, so the filter must keep tracking. Third, whenever the near-end user talks, s is in the error signal, and a filter that adapts on it will start cancelling the user. Every component below exists to deal with one of those three facts, under a budget of a few milliseconds of compute per 10 ms frame.

The reference and its clocks

The canceller can only remove what it can predict from its reference, so the reference must be the audio that actually reached the speaker. If the application feeds the canceller its own pre-mix TTS signal but the OS also plays a notification sound, mixes another app, or applies a volume curve, the echo contains audio the reference does not, and no filter can remove it. Use the platform's loopback or echo-reference path where one exists, or put the canceller at the last point in the render chain you control.

The second problem is clocks. The speaker is driven by a DAC clock and the microphone by an ADC clock. Both say 48 kHz, but they are rarely the same oscillator. A 50 ppm mismatch is 2.4 samples per second; over a ten-minute call the two streams drift about 30 ms apart. An adaptive filter sees that as an echo path that keeps sliding, so it never fully converges. Cancellers handle this by tracking the delay continuously and, when the two devices really are on separate clocks (a USB speaker with a built-in microphone elsewhere, say), resampling one stream to the other's rate.

Timestamp every render and capture buffer and keep the reference in a ring buffer indexed by presentation time. Most production echo bugs turn out to be timing bugs.

Delay estimation

The bulk delay between reference and capture is the sum of output buffering, acoustic flight time and input buffering. On phones and laptops it is often tens of milliseconds and sometimes more than a hundred, and it varies by device and audio route. The adaptive filter only covers a window of taps, so a separate estimator finds the bulk delay and the filter models what remains.

A robust estimator is generalised cross-correlation with phase transform (GCC-PHAT). Whitening the cross spectrum sharpens the correlation peak so that loud low frequencies do not smear it:

import numpy as np

def estimate_delay(ref, mic, max_lag):
    # Returns the lag (samples) by which mic trails ref, plus a peak-to-median confidence.
    n = 1 << int(np.ceil(np.log2(len(ref) + len(mic))))
    cross = np.fft.rfft(mic, n) * np.conj(np.fft.rfft(ref, n))
    cc = np.fft.irfft(cross / (np.abs(cross) + 1e-12), n)[:max_lag]
    lag = int(np.argmax(cc))
    confidence = cc[lag] / (np.median(np.abs(cc)) + 1e-12)
    return lag, confidence

Run it only when the reference has energy, and accept a new delay only when the confidence is high and the answer repeats over several windows. Jumping estimates usually mean a wrong reference or drifting clocks. WebRTC's AEC3 uses a bank of matched filters for the same job, which is cheaper to run continuously.

The linear stage: partitioned-block frequency-domain NLMS

A time-domain NLMS filter with thousands of taps is too slow to converge on coloured signals like speech, and too expensive. The standard answer is a partitioned-block frequency-domain adaptive filter: split the impulse response into P partitions of N taps, keep the FFT of each of the last P reference blocks, and do the convolution and the gradient in the frequency domain. Per-bin normalisation by the reference power makes the effective step size roughly even across frequencies, which is what speeds up convergence on speech. Partitioning keeps latency to one block instead of the whole filter length.

class PBFDAF:
    def __init__(self, block=64, partitions=50, mu=0.3, eps=1e-6):
        self.N, self.P, self.mu, self.eps = block, partitions, mu, eps
        bins = block + 1                                   # rfft of 2N samples
        self.W = np.zeros((partitions, bins), complex)     # filter, one row per partition
        self.X = np.zeros((partitions, bins), complex)     # reference spectra, newest first
        self.xbuf = np.zeros(2 * block)

    def process(self, x_blk, d_blk, step=1.0):
        N = self.N
        self.xbuf = np.concatenate([self.xbuf[N:], x_blk])
        self.X = np.roll(self.X, 1, axis=0)
        self.X[0] = np.fft.rfft(self.xbuf)
        y = np.fft.irfft((self.W * self.X).sum(axis=0))[N:]   # overlap-save output
        e = d_blk - y
        if step > 0:                                           # double-talk control sets step
            E = np.fft.rfft(np.concatenate([np.zeros(N), e]))
            norm = self.mu * step / (np.sum(np.abs(self.X) ** 2, axis=0) + self.eps)
            for p in range(self.P):
                g = np.fft.irfft(np.conj(self.X[p]) * E * norm)
                g[N:] = 0.0                                    # gradient constraint
                self.W[p] += np.fft.rfft(g)
        return e, y

Three details matter. The gradient constraint, zeroing the second half of each partition's update, stops circular-convolution artefacts from wrapping into the taps. The step argument is how the double-talk controller slows or freezes adaptation. And the update loop is the expensive part: two FFTs per partition per block. Many implementations update only a rotating subset of partitions each block, trading some convergence speed for a large cut in compute. Treat the code as a sketch; production cancellers add tuned step-size control and regularisation.

Double talk and divergence control

During double talk the error contains the user's voice, and a filter adapting at full speed drifts toward cancelling it. There are two families of defence.

Detect and freeze. Classic detectors compare levels (Geigel-style: is the microphone louder than the reference could explain?) or the coherence between reference and microphone, which drops when an independent talker is present. When they fire, set step near zero and keep subtracting with the current coefficients. Their weakness is the threshold: on a speakerphone the echo can be louder than the talker, so level-based detection misses quiet near-end speech.

Two filters. Run a background filter that adapts all the time and a foreground filter that produces the output. Copy background into foreground only when its error has been clearly smaller for some time, and reset the background from the foreground when it diverges. A double-talk burst can corrupt the background but never reaches the output. WebRTC's AEC3 uses a related two-filter design, with a refined filter and a faster coarse filter. Most robust designs combine both ideas: step-size control from echo-to-error ratios, plus a protected copy that can always be restored.

Residual echo suppression and comfort noise

A converged linear stage removes a large share of the echo, typically somewhere around 20-30 dB on good hardware, but not all of it. Small speakers driven hard distort, plastic chassis rattle, and the echo path changes faster than the filter tracks. What remains is residual echo, and it is loud enough to hear.

The residual echo suppressor estimates, per frequency band, how much of e is still echo, usually from the linear filter's own prediction y scaled by an estimated leakage factor, and applies a gain between 0 and 1 to each band. During far-end-only speech it can attenuate hard; during double talk it must back off, or the near-end voice gets chopped. Comfort noise fills suppressed bands at the background level so the line does not sound dead. Many current systems estimate the gains with a small neural network, which handles nonlinear residuals better. It still depends on a working linear stage and a correct reference.

Worked example: barge-in on a voice-agent kiosk

Take a voice-agent kiosk: one full-range speaker, a microphone 15 cm away, a reflective lobby, 16 kHz processing for the speech recogniser. The requirement is that a user can say "stop" while the agent is talking.

1. Reference. The TTS audio is mixed with UI sounds before playback, so the canceller taps the final mixed buffer, not the TTS stream, and timestamps it.
2. Filter sizing. The measured echo tail is about 200 ms, which is 3,200 taps at 16 kHz. With 64-sample blocks (4 ms) that is 50 partitions. Added algorithmic latency is one block.
3. Delay. GCC-PHAT reports a stable 38 ms bulk delay on this hardware, so the filter window starts there instead of wasting taps on silence.
4. Convergence. At start-up the agent plays a short greeting. The first second or so leaks some echo while the filter converges, so barge-in is disabled for the opening 1.5 s of the first utterance.
5. Barge-in gate. The recogniser's voice activity detector runs on e, not on the raw microphone. A barge-in fires only when near-end speech probability stays high for 250 ms while the suppressor's echo estimate in the speech bands is low. That rejects bursts of residual echo.
6. Result to check. Measure ERLE during agent-only playback, false barge-ins per hour of agent speech, and the delay from the user's first syllable to TTS stopping.

Telemetry and testing

The primary health metric is echo return loss enhancement: the ratio of microphone power to output power while only the far end is active. Compute it only in far-end-only frames, or near-end speech makes it meaningless.

def erle_db(d_blk, e_blk, far_active, near_active, state, alpha=0.98):
    # Smoothed ERLE over frames where only the far end is talking.
    if far_active and not near_active:
        state["pd"] = alpha * state.get("pd", 1e-9) + (1 - alpha) * float(np.mean(d_blk ** 2))
        state["pe"] = alpha * state.get("pe", 1e-9) + (1 - alpha) * float(np.mean(e_blk ** 2))
    return 10 * np.log10(state.get("pd", 1e-9) / state.get("pe", 1e-9))

Export it per session with the estimated delay, delay changes per minute, divergence resets and the audio route, split by device model.

Test offline with recorded reference and microphone pairs covering double talk, echo-path changes and drift, and keep the field recordings that once broke the canceller as a regression suite.

Failure modes

Trade-offs

• Filter length vs. cost and tracking. Longer filters cover longer tails but cost more and converge more slowly. Size them from measured tails, not guesses.
• Suppression vs. duplex. More residual suppression means cleaner far-end periods and worse interruptions.
• Platform AEC vs. your own. OS and browser cancellers know the real render path and are tuned per device, but they are a black box. Your own gives control and telemetry but needs the reference plumbing. Never run two cancellers in series without testing: the second sees a distorted, nonlinear signal.

Related reading

For the hardest hands-free case, read echo cancellation for speakerphones. For where AEC sits in the chain, see the WebRTC audio pipeline, noise suppression, voice activity detection and beamforming.

How does a real-time residual echo suppression filter compute its gains?

A real-time suppression filter works frame by frame on the same spectra as the linear stage. For each frequency bin it estimates how much of the error spectrum E is still echo, turns that into a gain between a floor and 1, smooths the gain over time, and multiplies E by it. It is causal and uses no lookahead, so it adds no latency beyond the block the linear stage already uses.

The residual estimate comes from the linear filter's own echo prediction Y. During far-end-only frames, learn a per-bin leakage factor: the smoothed ratio of |E| squared to |Y| squared, roughly the inverse of the linear stage's ERLE in that bin. Freeze it during double talk, exactly like the adaptive filter. The residual echo power is then the leakage times |Y| squared, scaled up by an overestimation factor because underestimating leaves audible echo.

def update_leak(leak, E, Y, far_only, alpha=0.95):
    # Per-bin residual-to-predicted echo power ratio, learned in far-end-only frames.
    if far_only:
        ratio = np.abs(E) ** 2 / (np.abs(Y) ** 2 + 1e-12)
        leak = alpha * leak + (1 - alpha) * np.minimum(ratio, 1.0)
    return leak

def suppressor_gains(E, Y, leak, g_prev, beta=2.0, g_min=0.05):
    residual = beta * leak * np.abs(Y) ** 2       # estimated echo left in E
    g = np.clip(1.0 - residual / (np.abs(E) ** 2 + 1e-12), g_min, 1.0)
    a = np.where(g < g_prev, 0.2, 0.7)           # fall fast, recover slowly
    return a * g_prev + (1 - a) * g

The output frame is the gain times E, with comfort noise added at the estimated background level in bins where the gain is low. The parameter values are illustrative starting points, not tuned figures.

The tuning knobs are trade-offs, not settings with a right answer:

• Overestimation (beta). Higher removes more echo and also more near-end speech during double talk.
• Gain floor (g_min). A floor above zero avoids musical noise and a line that sounds dead; a lower floor removes more echo.
• Fall and rise rates. Falling fast catches echo onsets. Rising slowly avoids flicker but clips the start of near-end words after the far end stops.

What to do next

1. Confirm your reference is the final rendered mix, and timestamp render and capture buffers.
2. Measure the bulk delay and the echo tail on each target device and route; size filter partitions from them.
3. Implement or enable continuous delay tracking and log delay changes per minute.
4. Add double-talk protection: a two-filter scheme or step-size control, ideally both.
5. Feed VAD, barge-in and ASR from the canceller output, and test that they never see the raw microphone.
6. Export ERLE in far-end-only frames, divergence resets and route per session, split by device model.
7. Build a recorded test corpus with double talk, path changes and drift, and run it on every build.