The core idea: decouple capacity from compute
Every dense transformer obeys one iron rule: the FLOPs it spends per token are roughly 2 × (number of parameters), because every weight is touched in the forward multiply-accumulate. Capacity and compute are welded together — you cannot buy more of one without paying for the other. Mixture-of-Experts is the architectural move that pries them apart.
The mechanism is conditional computation: instead of pushing every token through the same weights, the network chooses, per token, which subset of weights to use. Concretely, one big feed-forward layer becomes many smaller expert FFNs plus a router that activates only a few. Total parameters — the sum over all experts — set the model’s capacity to memorize patterns and specialize. Active parameters — the ones a given token actually runs — set the compute bill. Because only k of N experts fire, active can be a small fraction of total.
So the pitch is: get the representational richness of a huge model while paying, per token, only for a small one. The rest of this article is really just the bookkeeping and the caveats that make that pitch true — and the ways it can quietly fail.
The dense FFN we are replacing
To see what MoE changes, pin down what it replaces. In a standard transformer block, after attention, every token vector x of dimension d passes through a position-wise feed-forward network:
FFN(x) = W_2 · σ(W_1 · x + b_1) + b_2
W_1 : [d_ff, d] W_2 : [d, d_ff] d_ff ≈ 4dThe hidden width d_ff is conventionally about four times the model width d, and σ is a nonlinearity such as GELU or SwiGLU. This one block is where most of a transformer’s parameters live — typically around two-thirds of them, far more than attention’s projection matrices. Its per-token cost is roughly 2 · (d · d_ff + d_ff · d) = 4 · d · d_ff FLOPs.
Because the FFN dominates the parameter budget, it is the natural target for scaling capacity cheaply. If you want more parameters without more compute per token, you attack the FFN. MoE does exactly that: it keeps attention dense and shared, and shatters the single FFN into a set of interchangeable experts, each structurally identical to the block above.
From one FFN to N experts
An MoE layer instantiates N independent copies of the FFN — call them E_1, E_2, …, E_N — each with its own weights of the same shape as the dense block. Nothing about an individual expert is special; an expert is just a plain FFN. What is new is that we no longer run all of them.
The whole layer’s parameter count scales with N. If one dense FFN had P_ffn parameters, the expert bank has roughly N · P_ffn of them (plus the router’s tiny matrix). That is the capacity dial: an 8-expert layer stores 8× the feed-forward knowledge of its dense twin. Crucially, the experts sit in parallel, not in sequence — a token does not flow through all of them, it is routed to a chosen few.
Intuitively, experts get room to specialize. Different experts can come to handle different kinds of tokens — punctuation, numbers, code, a particular language, a syntactic role — though the specialization that emerges is learned and often less human-interpretable than that story suggests. The point is capacity: N experts give the model many more distinct transformations to draw on than a single shared FFN ever could.
The router: a softmax gating network
The router (or gate) is the brain of the layer, and it is astonishingly small: a single linear projection followed by a softmax. For a token x of dimension d:
logits = W_g · x W_g : [N, d] logits : [N]
g_i = softmax(logits)_i = exp(logits_i) / Σ_j exp(logits_j)The router weight matrix W_g maps the token into one score per expert; the softmax turns those N scores into a probability distribution g = (g_1, …, g_N) over the experts, with Σ_i g_i = 1. A high g_i means ‘this token belongs with expert i.’
Notice the router’s cost is trivial: W_g is just [N, d], so for N=8 and d=4096 it is about 32K parameters — nothing beside a single expert’s tens of millions. The router adds negligible compute and negligible parameters, yet it decides the entire computation path. That asymmetry — a tiny gate steering enormous expert banks — is what makes conditional computation cheap to add. The gate is trained end-to-end by ordinary backpropagation, learning to send tokens where they are handled best.
Top-k selection and the sparse combine
A softmax over all experts is still dense — every g_i is nonzero. Sparsity comes from keeping only the k largest gates and discarding the rest. This top-k step is what makes MoE sparse rather than a soft ensemble:
T = TopK(g, k) # indices of the k largest gates
w_i = g_i / Σ_{j in T} g_j for i in T # renormalize over the chosen k
y = Σ_{i in T} w_i · E_i(x) # run ONLY those k expertsOnly the k selected experts are ever evaluated; the other N−k contribute nothing and cost nothing. Their gate weights are renormalized so the combine is a convex weighted sum over the survivors. A common choice is k=2 (Mixtral, GShard); Switch Transformer pushes to the extreme k=1.
The gate value plays a double role: it selects and it scales. Keeping the weight w_i on the output (not just using it to pick) is what lets gradients flow back into the router — the expert’s contribution is multiplied by a differentiable w_i, so the loss can teach the gate to raise or lower that weight. The hard TopK itself is non-differentiable, but because the surviving weights are smooth, the router still learns which experts to prefer.
Total vs active parameters
Here is the definition the whole architecture turns on. Total parameters is everything the model stores: shared attention and embeddings, plus all N experts in every MoE layer. Active parameters is what a single token actually multiplies against: shared components plus only the k experts it was routed to.
total = shared + N · P_expert (per MoE layer: N experts stored)
active = shared + k · P_expert (per MoE layer: k experts run)
active / total ≈ k / N (for the expert-dominated part)With k=2 and N=8, the expert bank is exercised at only 2/8 = 25%. The forward FLOPs per token track the active count — roughly 2 × active — not the total. So compute is set by k, while capacity is set by N.
This is the decoupling made numeric. Raising N from 8 to 64 octuples the stored expert capacity while leaving per-token compute essentially unchanged (same k). You buy capacity in units of memory and buy compute in units of k — two separate purchases, which in a dense model were forced to be one.
Worked example: an 8-expert, top-2 model
Make it concrete with a Mixtral-style layout: hidden size d ≈ 4096, N = 8 experts per MoE layer, k = 2. Round numbers for intuition, not a datasheet:
Total parameters ≈ 47B (shared attention + 8 experts per layer)
Active per token ≈ 13B (shared attention + 2 of the 8 experts)
Forward FLOPs/token ≈ 2 × active ≈ 2 × 13B = 26 GFLOP
A DENSE 47B model ≈ 2 × 47B = 94 GFLOP per token
Compute ratio = 26 / 94 ≈ 0.28 (≈ k/N, as expected)Read the two axes separately. On memory, the model is a 47B model — every one of those 47 billion weights must sit in RAM or VRAM, because any token might route to any expert. On compute, each token costs about what a dense 13B model costs, roughly a 3.6× saving versus running all 47B.
The name ‘8x7B’ is a trap: it is not 8 × 7 = 56B, because attention and embeddings are shared, not replicated eight times. Only the FFN experts are octupled, which is why total lands near 47B and active near 13B — the quality of a large model at the running cost of a mid-size one.
FLOPs accounting, carefully
Why do FLOPs follow active parameters and not total? Because a multiply-accumulate only happens for weights that are actually used. An unrouted expert’s matrices are never loaded into the systolic array; they contribute zero operations for that token. The forward cost of an MoE layer per token is:
FLOPs_moe/token ≈ router (tiny) + k · (4 · d · d_ff)
≈ k · FLOPs_one_expertThe router term is negligible — 2·d·N for a layer with millions of expert FLOPs — so the whole layer costs about k dense FFNs, regardless of how many experts are stored. Add the shared attention cost (which every token pays in full) and you recover the 2 × active rule of thumb.
One honest asterisk: this is the arithmetic cost. Real MoE inference also pays for routing overhead — gathering the tokens assigned to each expert, dispatching them, and scattering results back — plus the memory traffic of loading whichever expert weights a batch touches. On hardware, that data movement, not the multiply-accumulates, is frequently the real bottleneck, which is a recurring theme once we get to CPUs.
Why routing collapses without help
Left to its own devices, the router tends to collapse. Early in training a few experts get slightly better by chance, so the gate sends them more tokens, so they improve faster, so the gate sends them even more — a rich-get-richer feedback loop. The endpoint is pathological: a handful of experts do all the work while the rest are starved of tokens, receive almost no gradient, and effectively die.
This wrecks the entire premise. You paid for N experts’ worth of memory, but if only two of eight are ever used, you have the capacity of a 2-expert model at the storage cost of an 8-expert one. Worse, dead experts are wasted parameters that can never recover, because a token never reaches them to generate a learning signal.
There is also a systems reason the imbalance hurts. In distributed training, experts are spread across devices, and computation proceeds in lockstep; an overloaded expert becomes a straggler that everyone waits on, while idle experts waste their hardware. So load balance is not a nicety — it is required both for the model to use its capacity and for the hardware to be utilized. That is why every practical MoE adds an explicit pressure toward balance.
The load-balancing auxiliary loss
The standard fix is an auxiliary loss added to the training objective that punishes uneven routing. Using the Switch Transformer formulation, for a batch of tokens and N experts, define two quantities per expert i:
f_i = fraction of tokens in the batch dispatched to expert i
P_i = average router probability g_i over the batch
L_aux = α · N · Σ_{i=1..N} f_i · P_iThe product f_i · P_i couples the hard assignment (a count, non-differentiable) with the soft probability (differentiable), so gradients flow through P_i and push the router toward a flatter distribution. The loss is minimized when load is uniform: at f_i = P_i = 1/N, the sum is N · (1/N)(1/N) = 1/N, giving L_aux = α — its floor.
The coefficient α (often around 0.01) sets how hard we insist on balance. Too small and experts collapse; too large and the router is bullied into ignoring token content, hurting quality. It is a genuine tension: the main loss wants the best expert per token, the auxiliary loss wants an even spread, and good MoE training lives at the compromise between them.
Capacity factor and token dropping
Balance in expectation is not the same as balance in a given batch. To make the computation fit fixed-size buffers on hardware, each expert is given a hard capacity — the maximum number of tokens it will accept from a batch:
capacity = capacity_factor · (tokens_per_batch · k / N)
capacity_factor = 1.0 → room for the exact average load
capacity_factor = 1.25 → 25% slack for uneven batchesThe term tokens · k / N is the average number of tokens an expert would receive under perfect balance; the capacity factor multiplies in a safety margin. If more tokens route to an expert than its capacity allows, the overflow tokens are dropped — that expert simply does not process them, and they pass through on the residual connection unchanged.
This is a direct compute-vs-quality knob. A larger capacity factor drops fewer tokens (better quality) but reserves more memory and compute and wastes any unused slots; a smaller one is leaner but drops more tokens, degrading their representations. At inference with small batches the effect is subtle, but during training the drop rate is watched closely — a high drop rate is a symptom that routing is imbalanced or the factor is set too tight.
Expert parallelism: where the experts live
MoE’s decoupling has a distributed-systems mirror image called expert parallelism. Because experts are independent, you can place different experts on different devices: expert 0 and 1 on GPU A, experts 2 and 3 on GPU B, and so on. Each device holds only its slice of the total parameters, which is how models with hundreds of billions or trillions of total parameters are trained at all.
The catch is that routing is now a network operation. After the gate decides assignments, tokens must be shipped to whichever device owns their chosen expert — an all-to-all communication — then the results shipped back. This all-to-all is the signature cost of expert parallelism, and it is why MoE scaling is often bounded by interconnect bandwidth rather than raw compute. Load imbalance makes it worse: a hot expert’s device becomes everyone’s bottleneck.
We flag this only as a pointer — it is a training and serving-cluster concern more than a math one — but it explains a design pressure you can feel elsewhere: keep k small, keep experts balanced, and prefer topologies where the all-to-all is cheap. The elegance of ‘just store more experts’ hides a very real communication bill.