Why it matters

Gradient clipping stabilizes training. Understanding shapes training safety.

Advertisement

The architecture

Global norm: clip if ||g|| > threshold.

Per-parameter: rarely used.

Gradient clipping flowCompute grad normL2 across all paramsComparevs thresholdScale gradientif over thresholdThreshold 1.0 typical; monitor clip rate to catch instability early
Gradient clip.
Advertisement

How it works end to end

Threshold: 1.0 typical.

g' = g * threshold / ||g|| if ||g|| > threshold.

Monitor clip rate: high = instability signal.