Why architecture matters here

MoE math failures are budget surprises: teams over-provision memory or under-provision experts. Architecture matters because sizing decisions at design time compound into every serving + training decision.

Advertisement

The architecture: every piece explained

The top strip is the parameter accounting. Total params = E × expert size + shared layers. Active params = K × expert + shared. Compute per token similar to dense at K/E ratio. Memory footprint dominated by total.

The middle row is capacity. Capacity factor — tokens per expert cap. Expert imbalance. Drop rate. Load balance loss.

The lower rows are practice. Effective ratio quality per FLOP. Scaling laws. Ops — sizing + hardware fit.

MoE math — expert count E + top-K + capacity factor + active vs total paramsthe parameters you pay for vs the parameters you computeTotal paramsE × expert size + sharedActive paramsK × expert + sharedCompute per tokensimilar to dense K/EMemory footprinttotal params dominateCapacity factortokens per expert capExpert imbalanceunder/over usedDrop ratetokens over capacityLoad balance losstraining regularizerEffective ratioquality per FLOPScaling lawsMoE vs denseOps — sizing E + K + capacity + hardware fitcapdetectmeasureregularizecompareextrapolateextrapolateoperateoperate
MoE math: total vs active params, capacity, load balance.
Advertisement

End-to-end flow

End-to-end: 8-expert model, top-2, capacity factor 1.25. Total = 8B, active = 2B. Compute ~= 2B dense but memory = 8B. Drop rate 0.4% at capacity 1.25. Load balance auxiliary loss keeps utilization even. Quality per FLOP beats dense 4B.