Gradient Clipping in Action

Gradient Clipping in Action: experiment guide

This simulation applies global L2 norm clipping to twelve explicit two-component gradient vectors. Each table row shows the input vector, original norm, common scale factor, clipped norm, and accumulated SGD parameter position. The initial parameter position is (0, 0). Both components are treated as parts of one global vector, so a single multiplier preserves its direction whenever the cap is positive. This differs from independently clamping coordinates. The inputs are a fixed teaching sequence, not gradients secretly generated by a training run, and the displayed positions do not imply that an unspecified loss function improves.

Worked details 1

Norm cap sets the largest permitted L2 magnitude and may be zero. Learning rate multiplies the clipped vector to produce an SGD update: the new position equals the previous position minus learning rate times clipped gradient. Spike multiplier changes only the sixth vector, originally (6, 8); all other input rows remain fixed so the effect of one outlier is easy to isolate. Reset parameters restores cap 1, learning rate 0.1, and spike multiplier 1. Any parameter edit pauses and restarts the sequence, ensuring that its cumulative positions are recalculated consistently from the origin rather than mixing settings from different experiments.

Worked details 2

Play reveals one update every five teaching seconds over a sixty-second timeline. Pause freezes the elapsed time and completed-update count. Restart returns to zero completed updates while preserving the current parameters. Speed provides quarter-speed, normal, and four-times-speed inspection; it affects only presentation. Seek moves directly to any instant and pauses. At zero seconds no update has happened; at five seconds row one is complete; at sixty seconds all twelve rows are complete. The chart draws the original and clipped magnitudes of completed rows on the same vertical scale. Its caption and table provide numerical alternatives, including the norm cap line, so the comparison does not depend on color vision. Playback is opt-in and initially paused.

Worked details 3

For an exact experiment, use the reset settings and seek to fifteen seconds. The first three vectors are (0.3, 0.4), (0.6, 0.8), and (3, 4), with norms 0.5, 1, and 5. Their scales are 1, 1, and 0.2. The position after those updates is (−0.15, −0.2). At twenty seconds the zero vector causes no movement and is assigned scale one, avoiding division by zero. Seek to thirty seconds to inspect the large sixth vector: its norm 10 becomes 1 using scale 0.1. Increasing Spike multiplier to 2 doubles the raw norm but leaves this clipped direction and update unchanged at cap 1. Raise the cap to 20 to see the unbounded update in that same row.

Worked details 4

Two boundary experiments clarify the limits. Set the cap to zero: all nonzero gradients scale to zero and no parameter moves. Set the learning rate to zero: gradients are still clipped and reported, but every SGD update is zero. Negative or nonfinite parameters are rejected. Norms exactly equal to the cap pass through unchanged. The mathematical model omits a numerical epsilon and uses ideal L2 scaling; specific optimizer libraries may add an epsilon or handle nonfinite gradients differently. Clipping constrains update magnitude for this SGD fixture, but it does not establish convergence, choose an appropriate cap for your network, or simulate Adam’s adaptive state.