Why architecture matters here
The naive framing of the problem — train a big model to output clean audio from noisy audio — collapses immediately against real-time constraints. A frame arrives every 10 ms; whatever processes it must be causal (no peeking at audio that has not happened), incremental (carry state, not re-read the whole utterance), and cheap (a few percent of one mobile core, because it runs for the whole call). These constraints eliminate most of the architectures that top offline benchmarks and force the design toward small recurrent or temporal-convolutional networks predicting masks over a compact band representation — RNNoise's insight of 22 Bark-scale bands driven by a GRU remains the archetype.
Masking is itself a load-bearing architectural choice. Estimating a bounded gain per band and multiplying it against the noisy spectrum means the system can only attenuate, never hallucinate: worst case, output approaches either silence or the unprocessed input. A generative model that resynthesizes the waveform can invent phonemes that were never spoken — an unacceptable failure class for calls and an actively dangerous one for anything downstream of transcription. Bounded-gain masking makes the failure modes graceful by construction.
Latency architecture matters as much as compute. Total algorithmic delay is roughly window length minus hop plus lookahead: a 20 ms window with 10 ms hop and no lookahead costs about 10 ms — fine inside a call chain whose mouth-to-ear budget is 150 ms. Every millisecond of lookahead buys quality (the model sees a sliver of the future before committing to a gain) and costs exactly that millisecond in delay; conferencing products typically spend 0-40 ms here, and the decision is a product decision, not an ML one.
The architecture: every piece explained
Top row: from air to features. Capture delivers PCM at 16 or 48 kHz — after the platform's echo canceller (AEC), which must run first, because suppressing noise before removing echo destroys the AEC's linear reference. Framing slices the stream into overlapping windows (20 ms window, 10 ms hop is standard) tapered by a Hann window so the STFT does not smear energy across bins. The STFT front-end produces a complex spectrum per frame; most systems compress it into 20-40 perceptual bands (Bark or ERB scale) plus a few pitch and temporal features — dimensionality the model can chew inside its budget, matched to how human hearing actually resolves frequency.
Right and middle: the estimator and the mask. The neural estimator is a causal GRU/LSTM stack or small TCN — tens to hundreds of thousands of parameters, not millions — whose recurrent state carries the noise context from frame to frame: what the espresso machine sounded like 300 ms ago is the evidence for suppressing it now. It outputs a gain per band in [0,1]. Two-stage designs like DTLN add a second network operating on the masked signal in a learned domain to clean residuals; DeepFilterNet upgrades scalar gains to short per-band FIR filters (deep filtering), which recovers detail plain magnitude masks blur.
The post-filter is the unglamorous stage that separates lab demos from shippable audio: temporal smoothing of gains (attack fast on speech onset, release slowly to avoid pumping), a spectral floor (never attenuate below, say, -25 dB, so the noise fades instead of gating unnaturally), and a VAD-driven override that relaxes suppression during confident speech to protect the voice. iSTFT overlap-add resynthesizes the waveform, using the noisy phase unaltered — phase estimation is expensive and magnitude masking with noisy phase is perceptually adequate at these frame sizes. Bottom row: the real-time budget (everything above must fit inside the hop, at p99, on the worst supported device) and the quality loop — DNSMOS/PESQ regression suites offline, human MOS panels and live A/Bs before rollout.
End-to-end flow
Steady state. A frame of 480 samples (10 ms at 48 kHz) arrives from the capture callback. It is appended to the analysis window, Hann-tapered, FFT'd; band energies and features are computed and fed to the model along with its carried recurrent state. Inference returns 30-odd gains and the updated state. The post-filter smooths the gains against the previous frame's, applies the floor, multiplies them into the spectrum, and the iFFT's output is overlap-added with the previous frame's tail. Clean samples leave the pipeline 10-30 ms after they entered the mic — a total cost of perhaps 1-3% of one core.
Noise onset. A door slams — broadband energy floods every band. The model's state has no history of this source, so the first frame or two pass partially suppressed; by frame three the recurrence has locked onto the profile and gains in non-speech bands drop toward the floor. This 20-30 ms adaptation is why transients are the hardest class and why training data curation (real impulsive noises, not just stationary loops) dominates transient performance.
Speech through noise. The user talks over the running dishwasher. Harmonic structure and temporal envelope push speech-band gains high while inter-harmonic and high-band noise stays clamped; the VAD signal simultaneously relaxes the post-filter's aggression to protect formants. The perceptual result — voice present, dishwasher a faint hiss — is the product working as designed, and it is worth noting what was sacrificed: some noise remains by policy (the spectral floor), because total removal sounds worse than mild residue.
Overrun. The device thermal-throttles mid-call and inference misses the hop deadline. A correct pipeline detects the overrun and degrades explicitly — skip the model and pass the frame with the previous gains, or drop to a cheaper classical fallback — rather than letting the audio callback underrun, which the user hears as glitching far worse than any noise.