Why it matters

Tensor Programs is scaling theory foundation. Understanding shapes research.

Advertisement

The architecture

Formal calculus of infinite-width limits.

Different parameterizations give different limits.

muP is one canonical.

Tensor Programs stackTensor programsformal frameworkInfinite-width limitscompute rigorouslyParameterizationsNTK vs muP vs SPYang 2019-2023 series; rigorous foundation for muP + feature learning
Tensor Programs.
Advertisement

How it works end to end

Yang 2019-2023 series.

NTK: lazy regime.

muP: feature learning.

Standard Parameterization: neither.