Why it matters

Kaplan started scaling law era. Historical + foundational understanding.

Advertisement

The architecture

L(N) ~ N^-alpha_N.

L(D) ~ D^-alpha_D.

L(C) ~ C^-alpha_C.

Under-provisioned data in their runs.

Kaplan scaling lawsParams NL ~ N^-alphaData DL ~ D^-alpha_DCompute CL ~ C^-alpha_CKaplan recommended more params, less data - Chinchilla later corrected
Kaplan.
Advertisement

How it works end to end

Power-law fits on log-log plots.

Alpha_N ~ 0.08, alpha_D ~ 0.11.

Compute-optimal: params > data heavy.