Why it matters

Shampoo is second-order optimizer for large models. Understanding shapes advanced training.

Advertisement

The architecture

Per-layer preconditioners (L, R matrices).

Root of preconditioners.

Update with preconditioned gradient.

Shampoo mathCompute L, Rper layerMatrix rootsL^-1/4, R^-1/4Preconditioned stepL^-1/4 G R^-1/4Distributed variants scale; used by Google for large runs
Shampoo.
Advertisement

How it works end to end

Gupta et al. 2018.

Preconditioner: outer product of gradients.

Fractional matrix power.

Distributed Shampoo scales.