Feed-Forward Expansion with Actual Activations

Feed-Forward Expansion with Actual Activations: experiment guide

This experiment computes a small dense feed-forward block instead of showing only tensor-shaped boxes. Input coordinate 0 and Input coordinate 1 specify the two components of one token vector. Hidden width selects an integer from two through eight. Shared hidden bias adds the same editable number to every hidden preactivation. Activation chooses ReLU or a linear ablation that removes the nonlinearity. The block always contracts back to two output coordinates. There is no residual addition, layer normalization, dropout or trained weight file. Changing any model input restarts the paused trace so that partially revealed rows never mix two configurations.

Worked details 1

Every weight is deterministic and visible. Number hidden units from j = 0. Their two up weights are (1 if j is even, otherwise −1) and (j modulo 3) − 1. Preactivation z is the dot product of those weights with the input plus the shared bias. ReLU gives h = max(0,z); the linear ablation gives h = z. Down weights are (j + 1)/width and (1 if j is even, otherwise −1)/width. Each row contributes h times these two down weights, and the output is the sum of all contributions. Parameter slots count two up weights, one hidden bias slot and two down weights per hidden unit: five times width. The bias slots are tied to one control in this fixture; the count is stored slots, not independent learnable degrees of freedom.

Worked details 2

Reset parameters restores input (1,2), width four, bias zero and ReLU. The four preactivations are −1, −1, 3 and −3. ReLU yields (0,0,3,0), so only hidden unit two contributes, producing output (2.25,0.75). Switch to Linear ablation and seek to sixty seconds: hidden values retain their signs and the output becomes (−1.5,1.5). This is a direct test that the activation changes the actual calculation. Return to ReLU and set bias to −10: every default hidden unit is inactive and the output becomes (0,0). A fractional hidden width is rejected instead of silently changing tensor shape.

Worked details 3

Play reveals hidden units in order and then performs the final output contraction over sixty teaching seconds. Pause freezes the trace. Restart clears progress while preserving parameters. Speed selects 0.25, 1 or 4 times the presentation rate, and Seek provides immediate paused inspection at any point. A hidden row completes every 60/(width + 1) seconds; the contraction appears at exactly sixty seconds. Signed bars show revealed hidden activations relative to the largest absolute activation, with exact weights and contributions in the table. All results are available as text, including negative and zero values. This is inspection time, never measured compute latency. Width changes both the number of units and the documented down-weight normalization, so it is not a controlled comparison of trained architectures or evidence that wider networks are better.