Why architecture matters here
The latency budget dictates everything. At 200ms glass-to-glass, you get roughly: 30ms capture and encode, 20ms packetization and send, 40-80ms network, 30-60ms jitter buffer, 20ms decode and render. Nothing in that chain may block, retry indefinitely, or wait for perfect information. TCP is disqualified at the first retransmission timeout, which is why WebRTC media runs over UDP (SRTP) and treats loss as a quality-degradation signal rather than something the transport hides. Once you accept loss as normal, the architecture must decide, per packet, whether repairing it (NACK retransmission) beats concealing it (decoder error concealment) beats resetting it (keyframe request) — a decision that depends on RTT, and thus must be made by machinery, not configuration.
The second forcing function is bandwidth uncertainty. A participant's available bandwidth can drop 10× in one second (WiFi to LTE handoff, a competing upload). The encoder must track a moving target the network never states explicitly — congestion is inferred from packet-arrival-time deltas via transport-wide congestion control (TWCC), and the target bitrate propagates back into encoder settings within a couple of RTTs. Systems that adapt slower than the network changes oscillate between freezing and wasting capacity.
The third is topology. Mesh (everyone sends to everyone) dies at 4-5 participants because uplink scales with N. The SFU architecture — each participant uploads once, the SFU forwards selectively — makes uplink O(1) and lets the server tailor what each receiver gets via simulcast (the sender encodes 2-3 resolutions simultaneously) or SVC (one layered bitstream). Every production video platform of consequence is an SFU fleet; understanding what the SFU can and cannot do (forward, drop, select — but never transcode in the hot path) explains most of its scaling behavior.
The architecture: every piece explained
Top row: the sender's media path. Capture pulls raw frames from the camera at a negotiated resolution and rate. The encoder compresses under real-time constraints — no B-frames (they need future frames, i.e., latency), tight rate control that hits the target bitrate per second not per file, and periodic reference structures so a lost frame doesn't corrupt indefinitely. The RTP packetizer slices encoded frames into MTU-sized packets stamped with sequence numbers and timestamps — the receiver's raw material for reordering and pacing. DTLS performs the handshake that derives keys, and SRTP encrypts every media packet; media is end-to-network-element encrypted by default, and with insertable frames / E2EE, end-to-end even through the SFU.
Second row: transport and control. ICE gathers candidate addresses — host, server-reflexive (via STUN, which discovers your public address), and relayed (via TURN, a server that forwards media when nothing else connects) — pairs them with the peer's, and connectivity-checks until a path works; roughly 10-20% of real traffic ends up relayed through TURN. Congestion control (TWCC): the receiver reports per-packet arrival times; the sender's bandwidth estimator reads queuing-delay trends and loss, computes a target bitrate, and drives the encoder — the loop that converts network state into pixels-per-second. The SFU receives each sender's simulcast layers and forwards, per receiver, the highest layer that fits that receiver's estimated downlink, switching layers without touching the codec. The repair toolkit: NACK asks for retransmission of specific packets (viable when RTT is low), FEC sends redundancy so losses are recovered without a round trip (viable when bandwidth is spare), and PLI requests a fresh keyframe when decode state is unrecoverable.
Bottom row: the receiver. The jitter buffer is the pipeline's shock absorber: it reorders out-of-order packets, holds frames just long enough to smooth network jitter, adapts its depth to measured variance, and decides when a missing packet has been waited-for long enough. The decoder reconstructs frames (concealing what's missing), and the renderer syncs to audio — lip-sync is maintained by delaying whichever stream is early. The ops strip is getStats: the in-browser telemetry (RTT, loss, jitter, bitrates, freeze counts, layer switches) that production monitoring is built on.
End-to-end flow
Follow one participant joining a 12-person call. Her app calls getUserMedia, negotiates SDP with the SFU (codecs, simulcast layers, extensions), and ICE begins: host candidates fail (symmetric NAT), STUN's server-reflexive candidate succeeds against the SFU's public address, and media flows after ~300ms of connectivity checks — no TURN needed this time. DTLS completes; SRTP keys are derived; her encoder starts three simulcast layers: 720p at 1.5 Mbps, 360p at 500 kbps, 180p at 150 kbps.
The SFU fans her packets out. A viewer on fiber gets the 720p layer; a viewer on hotel WiFi gets 360p; a phone that just backgrounded the app gets nothing (the SFU pauses forwarding — the cheapest packet is the one not sent). Each decision comes from the per-receiver bandwidth estimate the SFU maintains from its own TWCC loop with that receiver.
Two minutes in, her WiFi degrades: TWCC feedback shows inter-packet delay growing on her uplink. Her sender's estimator cuts the target from 2.15 Mbps to 700 kbps; the encoder drops the 720p layer entirely and reduces the 360p layer's frame rate — resolution and smoothness traded deliberately, per policy. Downstream, the SFU switches fiber-viewer from her (now absent) 720p to 360p at the next switch point. A burst of loss hits one receiver: its jitter buffer reports two missing packets; NACKs go out (RTT is 45ms, retransmission beats concealment); one packet returns in time, the other frame is concealed with barely a visible smear. Had the loss corrupted reference state, the receiver would have sent PLI, and the SFU — which caches the most recent keyframe per layer — answers from cache instead of asking her encoder, protecting all 11 other viewers from a keyframe storm.
She drives into a parking garage: bandwidth collapses, ICE detects the path failing, and an ICE restart re-gathers candidates — this time only TURN's relayed candidate works. Media resumes through the relay with ~40ms extra RTT; the estimator re-converges from a conservative floor; viewers see two seconds of frozen video and then 180p climbing back to 360p. The call survives because every stage was designed for exactly this.