A Mixture of Experts (MoE) layer answers a single, tempting question: can we grow a model’s parameter count — its capacity to store knowledge — without growing the compute we spend on every token? A dense transformer says no: every parameter participates in every forward pass, so more parameters means proportionally more FLOPs. MoE breaks that coupling. It replaces the feed-forward block with N parallel expert FFNs and a tiny router that, per token, picks only k of them to run. The model holds all N experts — that is its total capacity — but any one token only activates a k/N slice of them. The result is a model with the parameter count (and often the quality) of something far larger, at the compute cost of something far smaller. This piece works the math from first principles: the dense FFN it replaces, the softmax gating network, top-k selection and the sparse combine, the crucial distinction between total and active parameters, the load-balancing loss that keeps the router honest, capacity factors and token dropping, expert parallelism, and what the whole trade means when your inference target is a CPU with finite RAM.
The core idea: decouple capacity from compute
Every dense transformer obeys one iron rule: the FLOPs it spends per token are roughly 2 × (number of parameters), because every weight is touched in the forward multiply-accumulate. Capacity and compute are welded together — you cannot buy more of one without paying for the other. Mixture-of-Experts is the architectural move that pries them apart.
The mechanism is conditional computation: instead of pushing every token through the same weights, the network chooses, per token, which subset of weights to use. Concretely, one big feed-forward layer becomes many smaller expert FFNs plus a router that activates only a few. Total parameters — the sum over all experts — set the model’s capacity to memorize patterns and specialize. Active parameters — the ones a given token actually runs — set the compute bill. Because only k of N experts fire, active can be a small fraction of total.
So the pitch is: get the representational richness of a huge model while paying, per token, only for a small one. The rest of this article is really just the bookkeeping and the caveats that make that pitch true — and the ways it can quietly fail.
The dense FFN we are replacing
To see what MoE changes, pin down what it replaces. In a standard transformer block, after attention, every token vector x of dimension d passes through a position-wise feed-forward network:
FFN(x) = W_2 · σ(W_1 · x + b_1) + b_2
W_1 : [d_ff, d] W_2 : [d, d_ff] d_ff ≈ 4dThe hidden width d_ff is conventionally about four times the model width d, and σ is a nonlinearity such as GELU or SwiGLU. This one block is where most of a transformer’s parameters live — typically around two-thirds of them, far more than attention’s projection matrices. Its per-token cost is roughly 2 · (d · d_ff + d_ff · d) = 4 · d · d_ff FLOPs.
Because the FFN dominates the parameter budget, it is the natural target for scaling capacity cheaply. If you want more parameters without more compute per token, you attack the FFN. MoE does exactly that: it keeps attention dense and shared, and shatters the single FFN into a set of interchangeable experts, each structurally identical to the block above.
From one FFN to N experts
An MoE layer instantiates N independent copies of the FFN — call them E_1, E_2, …, E_N — each with its own weights of the same shape as the dense block. Nothing about an individual expert is special; an expert is just a plain FFN. What is new is that we no longer run all of them.
The whole layer’s parameter count scales with N. If one dense FFN had P_ffn parameters, the expert bank has roughly N · P_ffn of them (plus the router’s tiny matrix). That is the capacity dial: an 8-expert layer stores 8× the feed-forward knowledge of its dense twin. Crucially, the experts sit in parallel, not in sequence — a token does not flow through all of them, it is routed to a chosen few.
Intuitively, experts get room to specialize. Different experts can come to handle different kinds of tokens — punctuation, numbers, code, a particular language, a syntactic role — though the specialization that emerges is learned and often less human-interpretable than that story suggests. The point is capacity: N experts give the model many more distinct transformations to draw on than a single shared FFN ever could.
The router: a softmax gating network
The router (or gate) is the brain of the layer, and it is astonishingly small: a single linear projection followed by a softmax. For a token x of dimension d:
logits = W_g · x W_g : [N, d] logits : [N]
g_i = softmax(logits)_i = exp(logits_i) / Σ_j exp(logits_j)The router weight matrix W_g maps the token into one score per expert; the softmax turns those N scores into a probability distribution g = (g_1, …, g_N) over the experts, with Σ_i g_i = 1. A high g_i means ‘this token belongs with expert i.’
Notice the router’s cost is trivial: W_g is just [N, d], so for N=8 and d=4096 it is about 32K parameters — nothing beside a single expert’s tens of millions. The router adds negligible compute and negligible parameters, yet it decides the entire computation path. That asymmetry — a tiny gate steering enormous expert banks — is what makes conditional computation cheap to add. The gate is trained end-to-end by ordinary backpropagation, learning to send tokens where they are handled best.
Top-k selection and the sparse combine
A softmax over all experts is still dense — every g_i is nonzero. Sparsity comes from keeping only the k largest gates and discarding the rest. This top-k step is what makes MoE sparse rather than a soft ensemble:
T = TopK(g, k) # indices of the k largest gates
w_i = g_i / Σ_{j in T} g_j for i in T # renormalize over the chosen k
y = Σ_{i in T} w_i · E_i(x) # run ONLY those k expertsOnly the k selected experts are ever evaluated; the other N−k contribute nothing and cost nothing. Their gate weights are renormalized so the combine is a convex weighted sum over the survivors. A common choice is k=2 (Mixtral, GShard); Switch Transformer pushes to the extreme k=1.
The gate value plays a double role: it selects and it scales. Keeping the weight w_i on the output (not just using it to pick) is what lets gradients flow back into the router — the expert’s contribution is multiplied by a differentiable w_i, so the loss can teach the gate to raise or lower that weight. The hard TopK itself is non-differentiable, but because the surviving weights are smooth, the router still learns which experts to prefer.
Total vs active parameters
Here is the definition the whole architecture turns on. Total parameters is everything the model stores: shared attention and embeddings, plus all N experts in every MoE layer. Active parameters is what a single token actually multiplies against: shared components plus only the k experts it was routed to.
total = shared + N · P_expert (per MoE layer: N experts stored)
active = shared + k · P_expert (per MoE layer: k experts run)
active / total ≈ k / N (for the expert-dominated part)With k=2 and N=8, the expert bank is exercised at only 2/8 = 25%. The forward FLOPs per token track the active count — roughly 2 × active — not the total. So compute is set by k, while capacity is set by N.
This is the decoupling made numeric. Raising N from 8 to 64 octuples the stored expert capacity while leaving per-token compute essentially unchanged (same k). You buy capacity in units of memory and buy compute in units of k — two separate purchases, which in a dense model were forced to be one.
Worked example: an 8-expert, top-2 model
Make it concrete with a Mixtral-style layout: hidden size d ≈ 4096, N = 8 experts per MoE layer, k = 2. Round numbers for intuition, not a datasheet:
Total parameters ≈ 47B (shared attention + 8 experts per layer)
Active per token ≈ 13B (shared attention + 2 of the 8 experts)
Forward FLOPs/token ≈ 2 × active ≈ 2 × 13B = 26 GFLOP
A DENSE 47B model ≈ 2 × 47B = 94 GFLOP per token
Compute ratio = 26 / 94 ≈ 0.28 (≈ k/N, as expected)Read the two axes separately. On memory, the model is a 47B model — every one of those 47 billion weights must sit in RAM or VRAM, because any token might route to any expert. On compute, each token costs about what a dense 13B model costs, roughly a 3.6× saving versus running all 47B.
The name ‘8x7B’ is a trap: it is not 8 × 7 = 56B, because attention and embeddings are shared, not replicated eight times. Only the FFN experts are octupled, which is why total lands near 47B and active near 13B — the quality of a large model at the running cost of a mid-size one.
FLOPs accounting, carefully
Why do FLOPs follow active parameters and not total? Because a multiply-accumulate only happens for weights that are actually used. An unrouted expert’s matrices are never loaded into the systolic array; they contribute zero operations for that token. The forward cost of an MoE layer per token is:
FLOPs_moe/token ≈ router (tiny) + k · (4 · d · d_ff)
≈ k · FLOPs_one_expertThe router term is negligible — 2·d·N for a layer with millions of expert FLOPs — so the whole layer costs about k dense FFNs, regardless of how many experts are stored. Add the shared attention cost (which every token pays in full) and you recover the 2 × active rule of thumb.
One honest asterisk: this is the arithmetic cost. Real MoE inference also pays for routing overhead — gathering the tokens assigned to each expert, dispatching them, and scattering results back — plus the memory traffic of loading whichever expert weights a batch touches. On hardware, that data movement, not the multiply-accumulates, is frequently the real bottleneck, which is a recurring theme once we get to CPUs.
Why routing collapses without help
Left to its own devices, the router tends to collapse. Early in training a few experts get slightly better by chance, so the gate sends them more tokens, so they improve faster, so the gate sends them even more — a rich-get-richer feedback loop. The endpoint is pathological: a handful of experts do all the work while the rest are starved of tokens, receive almost no gradient, and effectively die.
This wrecks the entire premise. You paid for N experts’ worth of memory, but if only two of eight are ever used, you have the capacity of a 2-expert model at the storage cost of an 8-expert one. Worse, dead experts are wasted parameters that can never recover, because a token never reaches them to generate a learning signal.
There is also a systems reason the imbalance hurts. In distributed training, experts are spread across devices, and computation proceeds in lockstep; an overloaded expert becomes a straggler that everyone waits on, while idle experts waste their hardware. So load balance is not a nicety — it is required both for the model to use its capacity and for the hardware to be utilized. That is why every practical MoE adds an explicit pressure toward balance.
The load-balancing auxiliary loss
The standard fix is an auxiliary loss added to the training objective that punishes uneven routing. Using the Switch Transformer formulation, for a batch of tokens and N experts, define two quantities per expert i:
f_i = fraction of tokens in the batch dispatched to expert i
P_i = average router probability g_i over the batch
L_aux = α · N · Σ_{i=1..N} f_i · P_iThe product f_i · P_i couples the hard assignment (a count, non-differentiable) with the soft probability (differentiable), so gradients flow through P_i and push the router toward a flatter distribution. The loss is minimized when load is uniform: at f_i = P_i = 1/N, the sum is N · (1/N)(1/N) = 1/N, giving L_aux = α — its floor.
The coefficient α (often around 0.01) sets how hard we insist on balance. Too small and experts collapse; too large and the router is bullied into ignoring token content, hurting quality. It is a genuine tension: the main loss wants the best expert per token, the auxiliary loss wants an even spread, and good MoE training lives at the compromise between them.
Capacity factor and token dropping
Balance in expectation is not the same as balance in a given batch. To make the computation fit fixed-size buffers on hardware, each expert is given a hard capacity — the maximum number of tokens it will accept from a batch:
capacity = capacity_factor · (tokens_per_batch · k / N)
capacity_factor = 1.0 → room for the exact average load
capacity_factor = 1.25 → 25% slack for uneven batchesThe term tokens · k / N is the average number of tokens an expert would receive under perfect balance; the capacity factor multiplies in a safety margin. If more tokens route to an expert than its capacity allows, the overflow tokens are dropped — that expert simply does not process them, and they pass through on the residual connection unchanged.
This is a direct compute-vs-quality knob. A larger capacity factor drops fewer tokens (better quality) but reserves more memory and compute and wastes any unused slots; a smaller one is leaner but drops more tokens, degrading their representations. At inference with small batches the effect is subtle, but during training the drop rate is watched closely — a high drop rate is a symptom that routing is imbalanced or the factor is set too tight.
Expert parallelism: where the experts live
MoE’s decoupling has a distributed-systems mirror image called expert parallelism. Because experts are independent, you can place different experts on different devices: expert 0 and 1 on GPU A, experts 2 and 3 on GPU B, and so on. Each device holds only its slice of the total parameters, which is how models with hundreds of billions or trillions of total parameters are trained at all.
The catch is that routing is now a network operation. After the gate decides assignments, tokens must be shipped to whichever device owns their chosen expert — an all-to-all communication — then the results shipped back. This all-to-all is the signature cost of expert parallelism, and it is why MoE scaling is often bounded by interconnect bandwidth rather than raw compute. Load imbalance makes it worse: a hot expert’s device becomes everyone’s bottleneck.
We flag this only as a pointer — it is a training and serving-cluster concern more than a math one — but it explains a design pressure you can feel elsewhere: keep k small, keep experts balanced, and prefer topologies where the all-to-all is cheap. The elegance of ‘just store more experts’ hides a very real communication bill.
The memory-vs-compute trade, stated plainly
Strip MoE to its essential bargain and it is a trade of memory for compute. You spend memory lavishly — every expert must be resident, so total parameters set your RAM/VRAM footprint — in order to save compute, since only k/N of the experts run per token. When compute (or compute-bound latency) is your binding constraint and memory is comparatively cheap, that is a fantastic deal: you get big-model quality at small-model FLOPs.
The trade inverts the moment memory is the scarce resource. An MoE model is large to store even though it is cheap to run. You cannot keep only the ‘active’ 13B in memory, because the next token might route to any of the sleeping experts; you must hold all 47B. So MoE does not shrink the memory problem — it arguably worsens it — it shrinks the compute problem.
That framing predicts exactly where MoE shines and where it struggles. It shines when you can afford the parameters and want more quality per FLOP: large-scale training, and serving on memory-rich accelerators. It struggles wherever memory is tight relative to compute — which is precisely the situation of a small language model on a commodity CPU.
MoE on a CPU: the RAM wall
For the CPU-SLM setting this series cares about, MoE is a double-edged tool. The good edge is real: CPUs are compute-poor compared to GPUs, so an architecture that delivers strong quality at low per-token FLOPs is exactly what a CPU wants. A sparse 47B-total / 13B-active model runs its arithmetic like a 13B dense model — far friendlier to a CPU’s modest throughput than a dense 47B would be.
The bad edge is the RAM wall. Commodity machines have limited system memory, and MoE demands you hold all experts. A 47B model in 4-bit quantization is still around 24 GB just for weights; in fp16 it is closer to 94 GB — often more than the box has. The active-parameter saving buys you nothing here, because residency is dictated by total, not active.
And CPU inference is typically memory-bandwidth-bound: each generated token must stream its active expert weights from RAM through the caches. Because routing can pick different experts token to token, the working set thrashes and prefetching helps little. So on a CPU the honest read is: MoE trades away the compute you did not have much of, for memory pressure you can least afford. It can still win when total size fits comfortably in RAM — smaller expert banks, aggressive quantization — but the RAM budget, not the FLOPs, is the gate.
Pitfalls, subtleties, and honest caveats
A few things routinely trip people up. First, ‘active parameters’ is a compute claim, not a memory claim. Quoting a 13B active count next to a dense 13B model is fair on FLOPs and misleading on RAM; the MoE still weighs 47B on disk and in memory. Always report both numbers.
Second, MoE quality does not match a dense model of the same total size. A 47B MoE is generally weaker than a hypothetical dense 47B and stronger than a dense 13B — it lands between its total and active counts, closer to the geometric mean in practice. Sparsity is not free lunch; it is a favorable but lossy exchange.
Third, watch training instability: the router’s hard top-k makes the loss landscape jumpy, and load-balancing coefficients, capacity factors, and router-logit noise all need tuning. Fourth, the batch-size interaction: MoE loves large, diverse batches so every expert sees enough tokens to be efficient; at batch size one (a single-user CPU chatbot), you may load an expert’s full weights to process just one or two tokens, wrecking the efficiency the architecture was supposed to give. Understanding these keeps the elegant capacity/compute story from curdling into a disappointment in deployment.