MoE Routing Math

Why it matters

Routing shapes MoE quality + load. Understanding shapes design.

The architecture

Router logits: linear layer output.

Softmax over experts.

Top-K selection.

How it works end to end

Router: small MLP or single linear.

Top-K softmax.

Load-balance loss on aux.

Auxiliary z-loss for stability.