Why it matters

Width scaling is common lever. Understanding shapes design.

Advertisement

The architecture

Increase d_model.

Heads * head_dim = d_model.

muP: width-invariant HPs.

Width scalingd_modelhidden dimheads * head_dim= d_modelmuP width invariancecheap HP transferChinchilla-like frontier LLMs scale width + depth + data balanced
Width scaling.
Advertisement

How it works end to end

Params scale as d_model^2.

Attention memory: heads * seq^2.

muP enables transfer.