Feed-forward layers: matrices and gates

Configure a feed-forward layer

Inspect the hidden layer

Rows are computed independently. Scroll tables horizontally to inspect every numeric column.

Final outputs

Down-projection terms

Follow the actual projections

Each input row represents one token and each column one feature. Up weights map d input features into h hidden features; Down weights map h features back to d outputs. The same matrices apply independently to every token row. The page computes all matrix products and exposes every hidden value, final output and individual contribution to the down projection. Dimensions come from your matrices, including the actual h divided by d expansion ratio. No hidden units are silently capped or represented by decorative boxes.

Choose the computation

ReLU computes max(0, Input × Up) × Down. GELU uses the explicitly approximate tanh formula: one half times z times [1 + tanh(sqrt(2/pi) times (z + 0.044715 times z cubed))]. SwiGLU computes [SiLU(Input × Gate) elementwise multiplied by (Input × Up)] × Down. SiLU is z divided by (1 + exp(-z)), with beta fixed at one. The gate is a separate learned projection, not another name for SiLU alone. Its output is not a probability and can be negative or greater than one. All three variants omit biases in this experiment.

Use the editors and switches

Example selects Signed inputs, Zero gate or Wider hidden layer. Selecting its name alone preserves your current calculation; Load example copies all four matrices while preserving mode and the residual setting. Each JSON editor accepts one through eight rows and columns, rectangular arrays, and finite numeric entries from minus one hundred through one hundred. Up must be d by h, Down h by d, and Gate d by h. Gate is disabled and ignored in ReLU and GELU modes; its stored text becomes active and validated when you choose SwiGLU. Add input residual adds the original input to the branch output, without normalization. Reset defaults restores Signed inputs, ReLU and no residual. Invalid active inputs clear the numerical tables immediately.

Reproduce exact ReLU outputs

Reset defaults. The first input row is [1, -2], and its up projection is [1, -2, 3]. ReLU changes that to [1, 0, 3]. Multiplying by the displayed Down weights produces [4, -3]. The second input row [0, 1] produces [0, 1]. Turn on Add input residual: the final rows become [5, -5] and [0, 2], while the branch outputs stay unchanged. In Down-projection terms, the first output has contributions 1, 0 and 3. Their sum is exactly four; residual addition is separate. Edit only the second input row and verify every intermediate and output for the first row stays unchanged. This demonstrates position-wise computation without attention or cross-token mixing.

Test the gate and its failure boundary

Turn off the residual and select SwiGLU with the default matrices. The first gate projection is [1, -2, 0]. Its SiLU values are approximately [0.731059, -0.238406, 0]. Multiplication by the up projection gives [0.731059, 0.476812, 0], which also equals this row of final output in its first two coordinates. Load Zero gate while retaining SwiGLU: every branch output becomes zero despite nonzero input and up projection. Add the residual to recover exactly the input. Switch to ReLU and the branch becomes nonzero again because that mode does not use Gate. An invalid Gate editor is likewise ignored only while that control is inactive.

Inspect width and operation accounting

Load Wider hidden layer. Its input width is two and its hidden width four, so expansion is exactly two. In ReLU or GELU there are sixteen weight entries and sixteen scalar projection multiplications for its single token. SwiGLU uses twenty-four weights and twenty-four projection multiplications, plus four elementwise gate products. These counts exclude nonlinear functions, additions and the optional residual; they are not FLOP totals or measured hardware performance. Comparisons here keep the entered hidden width fixed. Matching parameter budgets across two-projection and three-projection architectures requires choosing different hidden widths; the page never implies that equal width means equal cost.

Read evidence within its limits

The hidden table distinguishes the up projection, gate projection, activated value and resulting hidden product. Final outputs separate the branch from residual addition. Down-projection terms expose every multiply before summation. Complete state evidence expands the full numerical record. Tables support horizontal keyboard scrolling on narrow screens. This is a direct numerical inspector, so there is no artificial playback duration. JavaScript double precision and rounded display values can differ in their last digits; the expanded record retains full computed precision. The small weights are teaching fixtures, not pretrained parameters. There is no training, bias, dropout, normalization, quantization, routing, attention, model-quality estimate or inference-speed claim.

Primary reference: GLU Variants Improve Transformer, equations for bias-free feed-forward layers and gated variants. This experiment uses hand-entered weights and makes no claim about trained-model quality.