Complete Transformer Block: Executable Tensor Trace

Complete Transformer Block: Executable Tensor Trace: experiment guide

This lesson executes a deliberately small transformer block on two tokens with two features each. First token coordinate x sets row zero to [x, 0]; Second token coordinate y sets row one to [0, y]. Both range from minus five to five. These are editable input features, not token IDs or learned embeddings. The block uses pre-normalization: normalize the input, compute attention, add the first residual, normalize again, compute a gated feed-forward branch, then add the second residual. Each node in the diagram has a corresponding computed tensor in the table. The full-precision inspector exposes intermediate values as JSON, so the diagram can be checked numerically rather than accepted as a sequence of labels.

Worked details 1

RMS epsilon is added to each row mean square before taking its square root. Each coordinate is divided by that denominator; the normalization gain is fixed to one. The opening epsilon is 0.00001. A zero vector with positive epsilon remains zero. Setting epsilon to zero when either input token is zero produces an explicit zero-denominator error and disables playback. No arbitrary normalized value is substituted. Branch gain multiplies both the attention output and the final feed-forward projection, from zero to two. This is an ablation control on two branch outputs, not the normalization gain or a learning rate. At gain zero and valid normalizers, the two residual additions preserve the input exactly.

Worked details 2

Attention uses one head. Q, K, V and the output projection are fixed two-by-two identity matrices, so the first normalized input supplies all three projections. Scores are pairwise dot products divided by the square root of two. Stable softmax subtracts the maximum permitted score before exponentiation. Causal attention, initially checked, forbids row zero from reading the second token; its probabilities are exactly [1, 0]. Disabling the checkbox lets both rows read both tokens. The second row is unaffected by this mask because it already sees both positions. No positional embedding, rotary transform, dropout, grouped-query head or KV cache is modeled. These limitations are deliberate and prevent a tiny fixture from being mistaken for a complete pretrained decoder.

Worked details 3

For the feed-forward branch, call the second normalized coordinates a and b. The fixed up projection is [a, b, a+b], while the gate projection is [a, b, a-b]. Apply SiLU to each gate coordinate, multiplying g by sigmoid(g), then multiply elementwise by the up vector. The hidden width is three. The down projection maps hidden values h0, h1, h2 to [h0+h2/2, h1-h2/2], followed by Branch gain. There are no biases or learned weights. The inspector exposes this hidden vector and the projected branch separately. This is actual gated arithmetic; it is not a claim that these fixed matrices perform useful language modeling.

Worked details 4

Reset parameters restores x=1, y=1, gain=1, epsilon=0.00001 and the causal mask. As an exact experiment, set gain to zero: the final output must be [[1,0],[0,1]], irrespective of the mask. Restore gain one and set x and y both to zero with positive epsilon: every activation and the output must be zero, while allowed attention rows remain normalized probability distributions. Set epsilon to zero next to trigger the failure boundary. After another reset, disable causal attention and compare output row zero with its previous value; row one must remain unchanged. Change x or y independently and inspect the corresponding scores and residuals rather than interpreting a color as an improvement.

Worked details 5

Play traverses eight computed stages over sixty teaching seconds. Pause freezes inspection. Next stage advances by one stage, Seek selects an inspection time while pausing, and Restart returns to the first stage without changing inputs. Speed offers quarter, normal and four times the presentation rate. Parameter edits recompute the entire trace and pause at the beginning; Reset parameters also restores the defaults. Full numerical tables remain available throughout playback, and the highlighted stage is additionally identified by text. Playback starts paused, including for reduced-motion users. The timing is pedagogical and has no relationship to CPU or GPU latency. The model uses JavaScript double-precision arithmetic and makes no BF16, instruction-count, training or benchmark claim.