When a mixture-of-experts layer is spread across GPUs, the interesting hardware event is not the expert matmul — it is the all-to-all that has to happen twice around it. Every token has to reach the rank that owns the expert the router picked, and every expert output has to come back. That is a different communication pattern from the all-reduce and all-gather of dense training: data-dependent, irregular, and at small batch sizes limited by how many messages you send rather than how many bytes. This piece treats the exchange itself as the object of study — the shape of its traffic, what makes it latency- rather than bandwidth-bound, how router skew deforms it, and how to find out whether it is actually costing you time.
Two collectives per layer, and what each one moves
An expert-parallel MoE layer contains two all-to-all exchanges, not one. Dispatch moves token activations from the rank holding the token to the rank holding its chosen expert. Combine moves the expert’s output vectors back along the inverse permutation, so each token is reassembled at its origin and scaled by the router’s gate weights.
Both move hidden-state vectors of the same width, so to first order dispatch and combine cost the same on the wire. With top-k routing each token is replicated k times on the way out and k partial results come back, so byte volume scales with k, not with the expert count. A third, much smaller exchange usually precedes dispatch: the per-destination counts, so every rank can size its receive buffer before the payload arrives. That metadata step is tiny in bytes and expensive in latency, because nothing else can start until it lands.
Why this is not an all-reduce
The collectives that dominate dense training are reductions: every rank contributes the same-shaped buffer, elements are combined arithmetically, and the algorithm is free to choose a route — ring, tree, or a hierarchy of both — because the operation is associative. Those algorithms are covered in the companion piece on NCCL collectives; the property that matters here is that they get to restructure the traffic.
All-to-all has no such freedom. Nothing is reduced; every rank has a distinct payload for every other rank, and it has to arrive at that specific destination. No intermediate rank can usefully combine two messages into one, so the exchange degenerates into a full mesh of point-to-point transfers. That structural fact drives everything below: message count cannot be cut by a smarter algorithm, only by aggregating where several destinations sit behind one link.
The message-size distribution is the whole story
Across an expert-parallel group of E ranks, one all-to-all is E × E logical messages: each rank posts a send and a receive for every peer. That count is fixed by the group size and does not shrink with the batch. Halving the number of tokens in flight halves the bytes per message but leaves the number of messages exactly where it was.
This is what makes the decode case counter-intuitive. A prefill step pushes thousands of tokens through the layer, so each message is a respectable contiguous buffer and the exchange behaves like a bandwidth problem. A decode step with a modest batch may hand each destination only a handful of token vectors — a hidden width in the low thousands at bf16 puts one token’s activation in the single-digit kilobytes — so you pay per-message overheads (descriptor posting, handshakes, completion polling) on a full mesh of transfers that each carry almost nothing. The link is idle; the queue is not.
Latency-bound at decode, bandwidth-bound at prefill
The same collective therefore sits in two different regimes depending on where you are in the request lifecycle, and tuning for one can hurt the other.
In the bandwidth regime the levers are familiar: send fewer bytes (lower-precision dispatch is common, since the activations feed a matmul that would cast them anyway) and keep buffers contiguous so the transport can stream. In the latency regime those levers do almost nothing — halving a message that was already too small to saturate anything changes little. What helps is reducing the number of independent transfers and of synchronisation points around them: aggregating destinations, fusing the permutation into the kernel that posts the sends, and avoiding a device-to-host round trip to read routing counts. Diagnose the regime first, or you will spend a week compressing payloads that were never the constraint.
Routing skew turns a symmetric collective asymmetric
An all-to-all is symmetric only if the router is uniform, and routers are not. Token-to-expert assignment is learned and data-dependent, so at any step some experts attract far more tokens than others. On the wire the E × E message matrix has wildly unequal entries: some rank pairs exchange a large buffer, others nearly nothing.
Because the layer cannot proceed until every rank has what it is owed, the exchange completes at the pace of the busiest pair. Ranks that drew a light assignment finish early and wait, and that wait is charged to the collective even though the fabric was idle for most of it. Two things follow. Average bandwidth utilisation is a misleading health metric — a skewed exchange looks under-utilised while being the critical path. And the mitigations that bound skew algorithmically (balancing losses, per-expert token budgets, overflow policy) are training- and serving-policy questions covered elsewhere; on the wire, skew shows up simply as sync time.
Overlapping dispatch with expert compute — a narrow window
The obvious optimisation is to hide the exchange behind arithmetic, and MoE makes that unusually hard. A rank cannot start its expert GEMM until that expert’s tokens arrive, and cannot start the combine until the GEMM produces output. Dispatch, compute, and combine form a strict dependency chain, so by default there is nothing local to overlap with.
The way out is to break the layer into chunks — expert groups, or slices of the token batch — and software-pipeline them, so chunk i’s GEMM runs on one stream while chunk i+1’s dispatch is in flight on another. The window that buys you is bounded by one chunk’s expert compute time, and chunking makes each transfer smaller — in the latency regime, exactly the wrong direction. At decode, where each expert may see only a few tokens, there is barely any compute to hide behind and the overlap evaporates. It is a real win during prefill and training; treat it as a bonus, not a plan, at low batch.
Bucketing: cut the message count, not the byte count
Since message count is what hurts, the central implementation trick is making each destination cost exactly one message: permute the token buffer so everything bound for a given rank is contiguous before any send is posted — sort by destination, build the offset table, then hand the transport one descriptor per peer rather than one per token.
Done naively the permutation is its own problem: gathering scattered rows into a new buffer costs a full read and write of the activations, which at decode can rival the transfer it was meant to accelerate. Production implementations fuse the permutation into the kernel that stages the sends, and keep routing indices and counts on-device so no host synchronisation is needed to size buffers. Libraries built for this pattern — DeepEP is the well-known example, layered on one-sided GPU-initiated primitives such as NVSHMEM — exist because the generic path posts too many small transfers and too many host round trips.
The node boundary is the design decision
Everything above is topology-agnostic, and topology is where the decision lives. Ranks inside one server talk over a high-bandwidth, low-latency intra-node interconnect; ranks in different servers talk through NICs and a switched fabric with far more latency and, typically, less bandwidth per GPU. An all-to-all that stays inside a node and one that crosses racks are not the same operation with different constants.
Read that figure’s banner as the floor, not the fix: a fast fabric is necessary and never sufficient. The structural remedy is hierarchical staging — gather everything bound for a remote node locally, send one aggregated transfer per node pair, then scatter inside the destination node. That converts a mesh over all ranks into a much smaller mesh over nodes: a message-count reduction, which is exactly the medicine the latency regime wants. The corollary is placement — if a token’s top-k experts sit inside one node, its dispatch never touches the NIC.
How to tell whether the all-to-all is your bottleneck
Start with a profiler timeline rather than a summary table. Communication kernels report a duration, but duration conflates two very different costs: time actually moving bytes, and time blocked waiting for a peer that has not posted its send. Long communication kernels with low link utilisation are almost always the second.
Three cheap experiments separate the cases. Replace the learned router with a uniform round-robin assignment: if the exchange gets dramatically faster, the problem is skew, not the wire. Run the same shape at a larger batch: if per-token cost falls steeply, you are latency-bound and should attack message count. Time a synthetic all-to-all at the same message sizes with no compute around it: that is the floor the fabric can deliver, and the gap between it and your measured layer time is permutation, launch, and synchronisation overhead you own in software.
A short checklist
When the exchange is slower than it should be, work down this list in order:
| Symptom | Likely cause | First move |
|---|---|---|
| Comm kernels long, links idle | Latency regime | Aggregate destinations; cut transfer count |
| Fast ranks idle at a barrier | Routing skew | Measure per-pair sizes before tuning transport |
| Cost jumps past one node | Crossing the NIC | Hierarchical staging; keep expert groups local |
| Gap vs. synthetic floor | Permute / launch overhead | Fuse permutation with staging; keep counts on device |
| Overlap not materialising | Nothing to hide behind | Check chunk compute time exceeds transfer time |
The through-line: MoE communication is a message-count problem dressed up as a bandwidth problem. Optimisations that assume the latter — compress the payload, buy a faster link — disappoint at exactly the batch sizes interactive serving cares about, because they attack a term that was already small.