Why it matters

SOAP is emerging second-order LLM optimizer. Understanding shapes trends.

Advertisement

The architecture

Shampoo-like preconditioners.

Adam-style adaptive step in preconditioned space.

SOAP structureShampoo precondL, R matricesAdam stepin precond spaceUpdate paramcombinedBetter generalization than pure Shampoo; faster than Adam+Shampoo separately
SOAP.
Advertisement

How it works end to end

Preconditioners: from Shampoo.

Adam-style m + v in precond space.

Sub-batch statistics.