Grokking — Delayed Generalization

Why it matters

Grokking illuminates learning dynamics. Understanding shapes deep learning.

The architecture

Phase 1: memorization.

Phase 2: sudden generalization.

Weight norm dynamics involved.

How it works end to end

Discovered by Power et al. 2021.

Long overfitting → sudden generalization.

Weight decay + training length key.

Not universal but real phenomenon.