Real-time noise suppression architecture, in depth: capture-chain order, band-gain masking, the per-frame loop and deadlines

Where suppression sits in the chain

Noise suppression is one stage in a chain, and its position matters. A typical voice capture path runs high-pass filtering, echo cancellation, noise suppression, automatic gain control, then the encoder. WebRTC's audio processing module follows this order. The echo canceller must come before the suppressor because it works by modelling a linear path from the far-end loudspeaker signal to the microphone. A suppressor that has already applied time-varying gains breaks that linear relationship, and echo leaks through. Gain control comes after, so it measures loudness on the cleaned signal and does not amplify noise during pauses.

The second positioning question is ownership. Operating systems, browsers and headsets often ship their own suppression. In a browser, getUserMedia accepts a noiseSuppression constraint. If you add your own model on top, you get double processing: two suppressors each remove a little speech along with the noise, and the artefacts compound. Decide which layer owns suppression, turn the others off explicitly, and verify through track.getSettings() that the constraint was actually honoured, since support varies by platform. The wider pipeline is described in the WebRTC audio pipeline.

Why masking over bands wins

Most deployed systems do not generate clean audio. They estimate a gain between 0 and 1 for each frequency region in each frame and multiply it into the noisy spectrum. That choice, called masking, is the most important architectural decision in the field, for two reasons.

First, it bounds the failure. If the model is wrong, the worst outcome is too much or too little attenuation, never an invented sound. A generative model that synthesises a waveform can produce speech-like output that was never spoken. That is an unacceptable failure for a phone call and a dangerous one in front of transcription. Second, gains are easy to post-process. You can smooth them over time, put a floor under them, or relax them when a voice detector is confident, all with simple arithmetic that the team can reason about.

Predicting a gain for every FFT bin is expensive: a 20 ms window at 48 kHz has 481 bins. So most designs group bins into perceptually spaced bands. Valin's RNNoise is the canonical example. It works on 48 kHz audio, summarises each frame in 22 bands instead of 480 complex spectral values, and feeds 42 features, including band-energy derivatives and pitch measurements, to three GRU layers that output 22 band gains. A pitch-based comb filter then removes noise between the harmonics of voiced speech, which band gains are too coarse to reach. Later designs refine the same pattern. DTLN chains two small LSTM stages, the second working in a learned transform domain. DeepFilterNet replaces scalar gains in low-frequency bands with short complex filters spanning several frames, recovering harmonic detail that magnitude masks blur.

The per-frame loop in code

The code below is the skeleton of the real-time loop with 10 ms hops at 48 kHz. The model is any causal network that takes one frame of features plus its hidden state and returns gains and a new state. Feature extraction is simplified to log band energies. Production systems add normalisation, pitch features and derivatives. The structure is what matters: one call per hop, state carried forward, and persistent buffers allocated once. NumPy still creates temporaries on every call, so a production version in C, C++ or Rust also writes the FFT and gain stages into preallocated scratch memory.

import numpy as np

SR, HOP, WIN = 48_000, 480, 960                      # 10 ms hop, 20 ms window
window = np.sqrt(np.hanning(WIN + 1)[:WIN])          # sqrt-Hann: analysis * synthesis sums to 1 at 50% overlap
# band_edges and interpolate_bands are small helpers you supply (bin-index tables)
edges = band_edges(n_bands=32, n_bins=WIN // 2 + 1)  # ERB- or Bark-spaced bin indices

class Suppressor:
    def __init__(self, model, floor_db=-25.0, attack=0.6, release=0.15):
        self.model, self.state = model, model.initial_state()
        self.inbuf = np.zeros(WIN, np.float32)
        self.outbuf = np.zeros(WIN, np.float32)
        self.g_prev = np.ones(len(edges) - 1, np.float32)
        self.floor = 10 ** (floor_db / 20)
        self.attack, self.release = attack, release

    def process(self, hop):                           # hop: 480 float32 samples
        self.inbuf[:-HOP] = self.inbuf[HOP:]; self.inbuf[-HOP:] = hop
        spec = np.fft.rfft(self.inbuf * window)
        energy = np.array([np.sum(np.abs(spec[a:b]) ** 2) for a, b in zip(edges, edges[1:])])
        feats = np.log10(energy + 1e-10)
        gains, self.state = self.model.step(feats, self.state)   # causal: this frame only
        # post-filter: rise fast on speech onsets, fall slowly to avoid pumping
        k = np.where(gains > self.g_prev, self.attack, self.release)
        g = self.g_prev + k * (gains - self.g_prev)
        g = np.maximum(g, self.floor); self.g_prev = g
        spec *= interpolate_bands(g, edges)           # band gains -> per-bin gains
        frame = np.fft.irfft(spec) * window
        self.outbuf[:-HOP] = self.outbuf[HOP:]; self.outbuf[-HOP:] = 0
        self.outbuf += frame
        return self.outbuf[:HOP].copy()               # finished samples

Three numbers are set here. The floor of about -25 dB stops the output from gating to digital silence between words, which listeners find more disturbing than faint residual noise. The asymmetric smoothing lets gains open within a frame or two when speech starts, so first syllables are not clipped, and closes them over many frames, so the background does not pump. The window choice fixes the delay floor: the first sample of each window cannot be output until the whole window has arrived, so latency is roughly the window length plus any lookahead frames the model uses.

Latency and CPU budgets

A 10 ms hop is a deadline, not an average. If any single call to process() takes longer than the hop, the audio device underruns and the listener hears a click or a gap. Budget for the worst case, measured on the slowest supported device with the thermal throttling that an hour-long call causes.

• Run on a real-time audio thread with no locks, no allocation, no logging and no garbage collector in the loop. Pre-allocate every buffer and scratch array, including the temporaries the NumPy skeleton leaves implicit.
• Size the model to the deadline. Shipping suppressors range from tens of thousands to around a million parameters. Measure real-time factor (processing time divided by audio time) at p99.9, not the mean.
• Watch for denormal floats. As recurrent state decays toward zero during silence, some CPUs slow down sharply on subnormal numbers. Enable flush-to-zero on the audio thread.
• Choose the lookahead deliberately. Each frame of lookahead improves onset handling and costs one hop of delay on every call. That is a product decision measured against your mouth-to-ear budget.

Training and measuring

Training data is synthesised because paired real recordings of the same speech with and without noise barely exist. Mix clean speech with noise clips at random signal-to-noise ratios, typically from about -5 dB to 25 dB. Convolve some mixes with room impulse responses so the model learns to handle reverberant speech, and randomise gain, microphone frequency response and clipping. The target for each band is the ideal gain: clean energy divided by noisy energy, square-rooted and capped at 1. The loss compares predicted and ideal gains, often with an extra penalty on attenuating speech, because users forgive residual noise far more readily than lost words. The Microsoft Deep Noise Suppression challenge datasets are a common starting point for clean speech and noise corpora.

Evaluation uses several metrics because none is sufficient on its own. PESQ and STOI compare output with a clean reference and estimate quality and intelligibility. DNSMOS predicts listener opinion scores without a reference, broken into speech quality, background quality and overall, following ITU-T P.835. Then add the metrics that matter for your product: word error rate when an ASR system consumes the output, real-time factor and deadline misses on target devices, and a small blind listening test before every release.

Worked example: a conferencing app

Consider a laptop conferencing app at 48 kHz with a mouth-to-ear budget of 150 ms, of which the network and jitter buffer use about 80 ms. The team chooses 20 ms windows, a 10 ms hop and no lookahead, so the suppressor adds about 20 ms. The model is a two-layer GRU over 32 bands, about 300,000 parameters, which runs at a real-time factor of 0.04 on the oldest supported laptop, leaving room for throttling.

In testing, DNSMOS background scores look excellent, but users complain that their voice sounds robotic when they type. Inspection shows gains collapsing on every keystroke and recovering slowly, so speech between keystrokes is attenuated. Raising the release coefficient helps, but the real fix is in the data. The training set had few examples of speech overlapping impulsive noise, so the team adds keyboard clips mixed during speech at low SNR and retrains. The next complaint comes from music teachers whose instruments are removed. The fix there is a product setting, not a model change: a toggle that raises the floor to 0 dB.

Failure modes

Trade-offs

Classical suppressors, which estimate a noise spectrum during pauses and apply spectral subtraction or a Wiener-style gain, cost almost nothing and handle stationary noise like fans well. They struggle with non-stationary noise such as keystrokes and dishes, which is exactly where neural models win. Larger neural models and longer lookahead improve quality but cost battery and delay. Generative and resynthesis approaches can sound cleaner still, but they trade bounded failure for the risk of invented content. Suppression also interacts with reverberation, which is a different problem handled in dereverberation. For most products, a small causal masking model with a careful post-filter is the right default, with a user-visible control for music and other intentional sound.

How do you deploy a real-time noise filter for BPO call centres?

In a BPO or contact centre, run one noise filter on the agent side, as close to the microphone as possible and before the softphone encoder, and treat the customer leg as a separate decision. The noise that matters most is not fans or traffic but neighbouring agents talking, and that is the hardest kind to remove.

Pick one owner per direction, as the double-processing rule above says. A softphone with its own suppressor plus a headset that also suppresses gives the muffled sound in the failure table.

Crosstalk and keyboards

A neighbouring agent is speech, so a general suppressor trained to keep speech keeps them too. What helps, in order: close-talking noise-cancelling boom microphones worn correctly, seating and acoustic panels, then speaker-aware models enrolled on each agent's own voice. Keyboard clicks are a training-data problem: include keystrokes overlapping speech, as in the worked example.

Latency and both directions

ITU-T G.114 treats up to about 150 ms one-way delay as acceptable for most conversations, and contact-centre calls already spend much of that on the carrier network, VoIP jitter buffers and sometimes a virtual desktop audio path. Each suppression stage adds roughly its window length, so two stages in series double the cost. Customer-side noise from cars and streets can be cleaned on the inbound leg at the media server, but many mobile phones already suppress, so keep it mild.

Measuring it

Score sampled recordings with DNSMOS per site, before and after, within your recording consent rules. Track ASR word error rate if calls feed analytics or QA. Run A/B by team and watch how often customers ask agents to repeat themselves.

What to do next

1. Draw your capture chain and confirm that echo cancellation runs before suppression and gain control after.
2. Decide which layer owns suppression, disable the rest, and verify the platform settings at runtime.
3. Fix the window, hop and lookahead from your end-to-end latency budget before choosing a model.
4. Measure p99.9 real-time factor on your slowest device during a long, warm session.
5. Build a test set covering keyboard noise during speech, babble, music and reverberant rooms, and track DNSMOS, ASR word error rate and speech attenuation per category.
6. Tune the attack, release and floor with listening tests, not only metrics.
7. Ship a user-facing setting for music and other sounds people want to keep.