Why architecture matters here
The SFU matters because it's the architecture that makes scalable multi-party video feasible -- avoiding the mesh's N^2/upload problem and the MCU's CPU cost -- and it's the foundation of essentially all modern video conferencing. Multi-party video is hard to scale, and the architecture determines whether it works. Mesh (every peer to every peer) fails at scale (N^2 connections, and each peer uploading its video N-1 times -- crippling the peers' uplinks). MCU (mixing all streams into one) works but is CPU-expensive (decoding all streams, compositing, re-encoding -- costly per call, limiting scale). The SFU (each peer uploads once, the SFU selectively forwards) balances both: it avoids the mesh's upload/N^2 problem (each peer uploads once to the SFU) and the MCU's CPU cost (forwarding streams, not decoding/re-encoding) -- so it scales multi-party video cost-effectively. This is why the SFU is the dominant architecture (used by essentially all modern video conferencing -- Zoom, Meet, and the like use SFU-based architectures). For anyone building or understanding scalable video conferencing, the SFU is the foundational architecture, and understanding it (why it beats mesh and MCU, how it adapts quality) is understanding scalable multi-party video.
The forward-not-mix insight is the architectural essence, and it's what gives the SFU its cost balance. The key distinction between the SFU and the MCU is what they do with the streams. The MCU mixes: it decodes all the incoming streams, composites them (into a single combined video -- e.g., a grid of participants), and re-encodes the result to send to each peer -- so each peer receives one mixed stream, but the MCU does expensive video processing (decode all, composite, re-encode -- CPU-intensive, costly per call). The SFU forwards: it receives each peer's stream and selectively forwards the streams (unmodified -- not decoded/re-encoded) to the other peers -- so each peer receives multiple streams (the other participants' -- rendered client-side), and the SFU does NO video processing (just forwarding packets -- cheap, scalable). This forward-not-mix approach is the SFU's essence: by forwarding (not mixing), the SFU avoids the MCU's expensive video processing (decode/composite/re-encode) -- so it's much cheaper per call (just packet forwarding -- scalable to many calls/participants). The tradeoff: the peers receive multiple streams (more download bandwidth and client-side rendering -- versus the MCU's single mixed stream) -- but this is usually favorable (client bandwidth and rendering are cheap; server CPU is the constraint). Understanding forward-not-mix (the SFU forwarding streams unmodified -- avoiding the MCU's expensive processing -- for scalability) is understanding the SFU's essence and its cost advantage.
And the simulcast-plus-selective-forwarding capability is what makes the SFU handle heterogeneous receivers, which is crucial for real multi-party calls. In a multi-party call, the receivers have different bandwidths (some on fast connections, some slow) and needs (the active speaker shown large, others small) -- so forwarding the same high-quality stream to everyone doesn't work (the slow receivers can't handle it). The SFU handles this with simulcast (or SVC): each peer sends multiple quality layers of its video (e.g., high, medium, low resolution -- simulcast sends separate encodings; SVC sends a layered encoding) upstream to the SFU. The SFU then selectively forwards the appropriate layer to each receiver based on their bandwidth and needs (a high-bandwidth receiver gets the high layer; a low-bandwidth receiver gets the low layer; a receiver showing a participant small gets a low layer) -- bandwidth adaptation per receiver. So the SFU adapts the quality per receiver (forwarding the right layer for each -- not one-size-fits-all) -- handling the heterogeneous receivers (different bandwidths, different needs). This -- simulcast/SVC (peers sending multiple layers) plus selective forwarding (the SFU choosing the right layer per receiver) -- is what makes the SFU handle real multi-party calls (with heterogeneous receivers), and understanding it is understanding how the SFU adapts quality (the key to good multi-party video experience across diverse receivers).
The architecture: every piece explained
Top row: the problem and comparison. The problem: mesh doesn't scale (every peer to every peer -- N^2 connections, each peer uploading its video N-1 times -- crippling uplinks). SFU: each peer uploads once to the SFU, which selectively forwards the streams to the others (forwarding, not mixing) -- scalable. vs MCU: the SFU forwards streams (no video processing -- cheap) vs the MCU mixes (decode/composite/re-encode -- CPU-expensive) -- the SFU's cost advantage. vs mesh: the SFU (each peer uploads once) vs mesh (each peer uploads N-1 times, N^2 connections) -- the SFU avoiding the mesh's upload/connection explosion.
Middle row: quality adaptation. Simulcast: each peer sends multiple quality layers (separate encodings -- high/medium/low) upstream -- so the SFU can choose which to forward per receiver. SVC (Scalable Video Coding): a layered encoding (a single stream with layers the SFU can selectively forward -- a base layer plus enhancement layers) -- an alternative to simulcast for layered quality. Bandwidth adaptation: the SFU forwards the right layer to each receiver based on their bandwidth (high-bandwidth receivers get high layers, low-bandwidth get low) -- adapting quality per receiver. Active speaker: prioritizing who matters (the active speaker -- forwarding their high-quality stream, and possibly lower quality for others -- so the important stream gets the bandwidth).
Bottom rows: scaling and control. Scaling SFUs: for very large calls -- cascaded SFUs (SFUs forwarding to each other -- distributing the load across multiple SFUs) and regional SFUs (SFUs in multiple regions -- receivers connecting to a nearby SFU, the SFUs cascading -- for geo-distributed calls) -- scaling beyond a single SFU. Congestion control: managing bandwidth -- REMB (Receiver Estimated Maximum Bitrate) and transport-cc (transport-wide congestion control -- estimating available bandwidth and adapting the sending rate) -- so the streams adapt to the available bandwidth (avoiding congestion). The ops strip: capacity (provisioning SFU capacity -- the SFU forwarding all the streams' packets, so it needs bandwidth and some CPU for the forwarding -- sized for the call load), quality adaptation (managing the quality adaptation -- simulcast/SVC layers, bandwidth adaptation, active-speaker prioritization -- for a good experience across receivers), and monitoring (monitoring the call quality -- bandwidth, packet loss, layer selection, receiver experience -- for quality and troubleshooting).
End-to-end flow
Trace an SFU handling a multi-party call. Five participants join a video call via an SFU. Each participant sends their video once upstream to the SFU (with simulcast -- each sending high, medium, and low quality layers). The SFU receives these five upstreams. For each participant (receiver), the SFU selectively forwards the other four participants' streams -- choosing the appropriate quality layer for each based on the receiver's bandwidth and needs (e.g., for a receiver on a fast connection showing the active speaker large and others small: the active speaker's high layer, the others' low layers). So each participant uploads once (to the SFU) and downloads the others' streams (at appropriate qualities) -- the SFU handling the distribution (selective forwarding). The SFU does no video processing (just forwarding the selected layers -- cheap, unlike an MCU's mixing), and each peer uploads once (unlike mesh's N-1 uploads). The call scales (the SFU forwarding, each peer uploading once) with adapted quality (the right layers per receiver) -- the SFU's scalable, quality-adaptive multi-party video.
The simulcast-adaptation and active-speaker vignettes show the quality handling. A simulcast-adaptation case: one receiver is on a slow connection (limited download bandwidth). The SFU forwards them the low-quality layers of the other participants (not the high layers -- which their connection couldn't handle) -- so they get a working call (lower quality, but smooth) adapted to their bandwidth. A high-bandwidth receiver gets the high layers (better quality). The SFU adapted the quality per receiver (simulcast layers forwarded per each receiver's bandwidth) -- handling the heterogeneous receivers. An active-speaker case: the SFU detects the active speaker (who's currently talking) and prioritizes their stream -- forwarding the active speaker's high-quality layer to receivers (so the person talking is shown clearly), while the non-speaking participants may be forwarded at lower quality (they're shown small) -- the active-speaker prioritization directing the bandwidth to who matters (the speaker).
The scaling and congestion vignettes complete it. A scaling case: a very large call (hundreds of participants) exceeds a single SFU's capacity -- so the team uses cascaded SFUs (multiple SFUs, forwarding to each other -- distributing the participants across the SFUs, the SFUs cascading the streams between them) and regional SFUs (participants connecting to a nearby SFU -- lower latency -- the regional SFUs cascading) -- scaling the call beyond a single SFU. A congestion case: a receiver's available bandwidth drops (network congestion). The congestion control (transport-cc -- estimating the reduced bandwidth) signals it, and the SFU adapts (forwarding lower-quality layers to that receiver -- matching the reduced bandwidth) -- avoiding congestion (the streams adapting to the available bandwidth). The consolidated discipline the team documents: use an SFU for scalable multi-party video (each peer uploads once, the SFU selectively forwards -- avoiding the mesh's N^2/upload problem and the MCU's CPU cost), leverage forward-not-mix (the SFU forwarding streams unmodified -- cheap, scalable), use simulcast/SVC plus selective forwarding for quality adaptation (peers sending multiple layers, the SFU forwarding the right layer per receiver's bandwidth), prioritize the active speaker (directing bandwidth to who matters), scale with cascaded/regional SFUs for large calls, use congestion control (REMB, transport-cc -- adapting to available bandwidth), provision SFU capacity, and monitor call quality -- because the SFU is the architecture that makes scalable multi-party video feasible (forwarding, not mixing -- avoiding the mesh's and MCU's problems), with simulcast/SVC and selective forwarding adapting quality per receiver, foundational to modern video conferencing.