Why it matters

Lion challenges Adam. Understanding shapes optimizer choice.

Advertisement

The architecture

Update: sign of interpolated momentum.

1 momentum tensor (vs Adam's 2).

Lion optimizerInterpolate momentumbeta1Signof momentumUpdate paramwith weight decaySimpler than Adam; less memory; discovered by evolutionary search
Lion math.
Advertisement

How it works end to end

c = beta1 * m + (1-beta1) * g.

Update = sign(c).

m = beta2 * m + (1-beta2) * g.

Weight decay separately.