Why it matters
Grokking hints at generalization mechanisms. Research area with training implications.
Advertisement
The architecture
Memorize first.
Generalize suddenly later.
Weight decay drives.
Advertisement
How it works end to end
Power et al. 2022.
Weight decay matters.
Long training needed.