Why it matters

Switch Transformer established MoE at scale. Historical + design understanding.

Advertisement

The architecture

K = 1 routing.

Load balance loss.

Capacity factor.

Switch Transformer flowTokeninputRouter K=1single expertCompute + returncombinedSimplicity + scale; forerunner of Mixtral, DeepSeek, Grok
Switch.
Advertisement

How it works end to end

Fedus et al. 2021.

1.6T param model demonstrated scale.

Foundation for modern MoE.