Why it matters

Switch Transformer is top-1 MoE. Scaled to 1.6T params at Google.

Advertisement

The architecture

Top-1 routing.

Load balance loss.

Expert capacity.

Switch TransformerTop-1 routersingle expert per tokenLoad balanceauxiliary lossExpert capacitybuffer + dropFedus et al. 2021; 1.6T params; top-1 for simplicity
Switch Transformer.
Advertisement

How it works end to end

Fedus 2021.

1.6T Google.

Top-1 routing.