Why it matters

MoE scales params without proportional compute. Standard for scale.

Advertisement

The architecture

Router picks top-k experts.

Load balance loss.

Sparse activation.

Mixture of expertsRoutertop-k expertsSparse activationonly k activeLoad balanceauxiliary lossFedus et al. Switch Transformer; Mixtral 8x7B, DeepSeek V3 use MoE
Mixture of experts.
Advertisement

How it works end to end

Switch Transformer 2021.

Mixtral 8x7B.

DeepSeek V3.