Regularization — Preventing Overfitting Through Constraints

Why it matters

Every training pipeline uses regularization. Understanding the techniques and when to use each shapes model quality.

The architecture

L2 (weight decay): add penalty proportional to sum of squared weights. Encourages small weights.

L1: penalty on absolute values. Encourages sparse weights.

How it works end to end

Dropout: randomly zero fraction of neurons during training. Prevents co-adaptation.

Data augmentation: transform training data (rotate, crop, translate for images; back-translate for text). Effectively more data.

Batch norm: normalizes activations. Has regularizing side-effect.

Early stopping: stop training when val loss stops improving. Simplest and often most effective.