Why architecture matters here
Architecture matters here because a mixer sits on the critical path of a live conversation, and every millisecond and every arithmetic decision it makes is audible. Unlike a batch audio job, there is no second take: the samples flow through once, in order, under a deadline, and any mistake — a drifted clock, a clipped sum, a listener's own voice leaking back — is heard by real people in real time. The mixer is where several independent, imperfect network streams have to be reconciled into one coherent sound, and the structure of how you do that reconciliation determines whether the call sounds like a room or like a fault.
The first structural reason mixing is worth the cost is the collapse from N streams to one. Without a mixer, an endpoint in an M-party call must receive M-1 streams and mix them itself; the downstream bandwidth, the number of decoders, and the CPU all grow with the room. That is fine for a beefy desktop in a small call and ruinous for a phone in a large one. The mixer amortizes the combining work once on the server so every client, however weak, sees exactly one inbound stream. This is what lets a fifty-person town hall include a caller on a decade-old handset.
The second reason is heterogeneity. Real conferences mix WebRTC browsers, mobile apps, and PSTN callers arriving through a gateway, and those endpoints speak different codecs, sample rates, and packet cadences. A mixer is the natural place to normalize all of that: decode everything to a common internal representation, do the signal processing once, and re-encode to whatever each output leg needs. A forwarding architecture cannot do this — it just relays opaque packets — which is why any call that must bridge to the telephone network ends up with a mixer somewhere, because the phone network wants one mono stream, not a bundle.
The third reason is control over the sound itself. Because the mixer has every stream decoded to raw samples at once, it can apply cross-stream intelligence that a forwarder cannot: automatic gain to bring a quiet talker up to a loud one, a limiter to keep the sum from clipping when three people laugh together, ducking to attenuate background talkers behind the active speaker, and active-speaker selection so that a room of fifty does not sum fifty streams of mostly silence and hiss. These are room-level acoustic decisions that are only possible when one component sees all the audio simultaneously.
But every one of those benefits is bought with latency and CPU, and that is the tension the architecture must manage. Each stream must pass a jitter buffer, which adds delay by design; decode and re-encode add more; and the summation is a synchronization point that cannot proceed until enough of each stream has arrived. Meanwhile the server does N decodes and up to N encodes per room per frame, so cost is quadratic-ish in the worst case and dominates the media plane's compute bill. A well-built mixer is a continuous negotiation between fidelity and these two budgets — add just enough buffering to smooth jitter without adding audible delay, mix just enough streams to sound complete without wasting encodes on silence. Get that balance wrong and you have either a laggy call or an expensive one, and often both.
The architecture: every piece explained
The pipeline begins on the left with the inbound legs. Each participant sends encoded audio — Opus from a browser, AAC or G.711 from other endpoints — as a stream of RTP packets. These packets do not arrive on a tidy schedule; they are jittered, sometimes reordered, and sometimes lost. So the very first stage each stream hits is a per-stream jitter buffer that holds a small backlog, reorders by sequence number, and hands frames to the decoder on a smooth cadence. When a packet is missing, the decoder's packet-loss concealment (PLC) synthesizes a plausible fill so the mix does not get a hole. Every stream gets its own buffer because every stream has its own network path and its own jitter profile.
After decoding, the samples must be brought to a common clock and sample rate — typically 48 kHz internally. This is the resampling and drift-correction stage, and it is more important than it looks. Each participant's audio hardware has its own crystal oscillator, so a stream nominally at 48 kHz might really be at 48,001 or 47,998 Hz; over minutes those tiny differences accumulate into whole samples of drift that, if uncorrected, cause periodic glitches or a slow desync between streams. The mixer resamples each stream to its own master clock so that when it sums them, sample N of every stream corresponds to the same instant of wall-clock time.
With every stream decoded and aligned, the active-speaker selection stage decides which streams actually go into the mix. In a large room most participants are silent most of the time, and summing fifty mostly-silent streams just piles up their noise floors into a hiss and wastes CPU. So the mixer measures each stream's energy and mixes only the loudest few — commonly the top three to five talkers — while muting the rest. This both cleans up the sound and bounds the summation cost: the mix is over a small constant number of streams no matter how big the room grows.
The selected streams flow into the summation stage, which is where the actual mixing arithmetic happens, wrapped in two protective controls. Automatic gain control (AGC) normalizes levels so a soft-spoken participant is not buried under a loud one. Then the samples are summed — and because the sum of several full-scale signals can exceed the maximum representable value, a limiter sits on the output to compress peaks smoothly rather than letting them clip into harsh distortion. The talk-over and ducking logic works alongside this: when there is a clear dominant speaker, background talkers are attenuated so the mix stays intelligible instead of dissolving into a crowd noise.
Finally the mixer produces output, and here is the piece that makes it N-way rather than one-way: mix-minus encoding. For each listener the mixer computes a mix of everyone except that listener, then encodes it to that leg's codec. A participant must never hear their own voice looped back through the bridge, because the round-trip delay turns it into echo, so the mixer subtracts each participant's own contribution from the mix it sends them. In practice this is done efficiently — sum everyone once, then subtract each listener's stream for their personal mix — but conceptually it is N distinct encodes per frame, which is why the encode count on the ops strip is the mixer's dominant cost. The strip also watches clip rate, active-speaker churn, and the latency the mixer adds to mouth-to-ear, because those three tell you whether the room sounds clean, stable, and responsive.
End-to-end flow
Follow a single talk spurt through the bridge in a six-person call. Participant B starts speaking. Their microphone samples are encoded to Opus and sent as RTP packets across the network to the mixer, arriving slightly jittered — some packets a few milliseconds early, one delayed, one briefly out of order. B's jitter buffer absorbs this: it holds a short backlog, puts the reordered packet back in sequence, and releases frames to the decoder at a steady 20-millisecond cadence. The decoder turns each frame into raw PCM samples, and when the one delayed packet has not arrived in time, PLC synthesizes a brief fill so the decoded stream has no gap.
B's decoded audio is resampled onto the mixer's master 48 kHz clock, correcting for the fact that B's device clock runs a hair fast. Now B's samples are time-aligned with everyone else's. The active-speaker selector measures the energy of all six streams; B is clearly talking, two others are silent, and one is rustling papers at low level. The selector picks B as the dominant speaker and includes maybe one other low-level stream, muting the rest so their noise floors never enter the mix.
The selected streams reach the summation stage. AGC notices B is speaking a little quietly and lifts their gain toward a target loudness. The samples are summed; at one moment B and another participant both peak, and the running sum briefly exceeds full scale — the limiter catches this and gently compresses the peak instead of allowing a clipped, crackling sample. The ducking logic, seeing B as the clear dominant talker, attenuates the paper-rustling background so B stays crisp. The result is a single clean 48 kHz mix of the room's active audio.
Now comes the personalization. For participants A, C, D, E, and F the mixer sends a stream containing B's voice — but for B's own leg, the mixer computes a mix-minus that excludes B. Since B is currently the only real talker, B's mix-minus is essentially silence plus a touch of the low background, which is exactly right: B hears the room without hearing themselves. Each listener's mix is then encoded to their leg's codec — Opus for the browsers, G.711 for the phone caller bridged in through the gateway — and sent back out as RTP. Six inbound streams became one clean summed mix and up to six personalized, re-encoded outbound streams, all within a frame time.
Step back and count the latency B's voice accumulated: network transit in, the jitter buffer's deliberate hold, decode, resample, the summation frame boundary, encode, and network transit out. Each stage is small, but they add up, and the sum is the mouth-to-ear delay the other five participants experience. The whole art of the mixer is keeping that total under the threshold where conversation feels natural while still smoothing enough jitter to avoid glitches — and doing it for every simultaneous room on the server, each costing its bank of decodes and mix-minus encodes every frame. That is why the mixer's design is a permanent trade between how good it sounds and how much it costs to run.