Backpropagation: the Chain Rule

Backpropagation: the Chain Rule: experiment guide

This lesson computes an actual scalar network: z = w × x + b, h = max(0, z), prediction = v × h, and loss = (prediction − target) squared divided by two. Input x, first weight w, bias b, output weight v, and Target each change the forward result and its derivatives. The numbers are a deliberately small fixture, not weights from a trained transformer. Reset parameters restores x = 2, w = 0.5, b = 0.3, v = 0.7, and target = 1.5. Changing a parameter pauses and restarts the trace so that cached derivatives from the previous graph cannot appear beside new activations.

Worked details 1

Play reveals the computation over a sixty-second teaching timeline. The first half walks through the input, affine transform, ReLU, prediction, and loss; the second half follows derivatives in reverse order. Pause freezes the current stage. Restart returns to the first stage and pauses without changing parameters. Speed chooses one quarter, one, or four times the teaching playback rate; it changes neither the network nor its arithmetic. Seek selects any point from zero to sixty seconds and pauses for inspection. Forward pass jumps to the end of the forward calculation at thirty seconds, while Backward pass reveals the complete calculation at sixty seconds. These actions are inspection shortcuts rather than overlapping asynchronous computations. Playback begins paused, including when reduced motion is requested.

Worked details 2

At the reset values, z and h are both 1.3, prediction is 0.91, error is −0.59, and loss is 0.17405. The derivative at the prediction is −0.59. Multiplication by v gives dL/dh = −0.413. Because z is positive, the ReLU local derivative is one and dL/dz remains −0.413. The displayed parameter gradients are dL/dw = −0.826, dL/db = −0.413, and dL/dv = −0.767. The input derivative is dL/dx = −0.2065. Pause at successive backward stages and multiply the displayed upstream derivative by the relevant local factor before revealing the answer. Values in the trace table are available as ordinary text, so color and arrows are not required to follow the calculation.

Worked details 3

To test a failure boundary, change b to −2 while leaving the other reset values unchanged. Then z = −1, h = 0, prediction = 0, and loss = 1.125. The upstream derivative dL/dh is −1.05, but the inactive ReLU blocks it: dL/dz, dL/dw, dL/db, dL/dx, and dL/dv are all zero. A nonzero loss therefore does not guarantee a useful parameter gradient in this graph. Set b to −1 to reach z = 0 exactly. ReLU is nondifferentiable there; this fixture explicitly chooses derivative zero, rather than claiming a unique mathematical derivative. Away from that kink, central finite differences independently check each analytic derivative in the model tests. The lesson has no optimizer, tensors, dropout, or attention, and its presentation seconds are not compute latency.