Why architecture matters here
Separation fails on artifact leak, streaming latency, and generalization gaps. Architecture matters because model + latency + eval decide usefulness.
The architecture: every piece explained
The top strip is the pipeline. Mixed input waveform. STFT / spectrogram TF representation. Neural model — U-Net or Conv-TasNet. Masks per source elementwise.
The middle row is variants. Inverse STFT reconstructs. Streaming causal for low-latency. Speaker / instrument targeting via conditioning. Metrics — SI-SDR + MOS.
The lower rows are practice. Use cases — conferencing + music + accessibility. Failure modes — leaks + artifacts. Ops — deploy + latency + eval.
End-to-end flow
End-to-end: conference audio with 2 speakers. Streaming causal model separates. Each speaker heard cleanly. Latency 40ms. SI-SDR 12 dB. Deployed to production; conferencing quality metric climbs.