Why it matters

Init determines training stability. Understanding shapes design.

Advertisement

The architecture

Xavier: sqrt(2 / (fan_in + fan_out)).

Kaiming: sqrt(2 / fan_in) for ReLU.

Modern LLM: small std (0.02) for many.

Init variantsXavier / Glorotlinear activationsKaiming / HeReLU / SiLULLM std=0.02empirical modernGPT-2 uses std=0.02; muP rescales further; init interacts with optimizer + LR
Init scaling.
Advertisement

How it works end to end

Preserve variance forward + backward.

LLM: often std = 0.02 empirically.

muP: rescale further.

Special for residual: down-scale output projection.