Multi-head attention: projections and masks

Configure self-attention

Inspect each head

Scroll each numeric table horizontally to inspect every column. Tables also support keyboard focus and scrolling.

Combine all heads

Trace real attention products

Input contains one feature vector per token. Query, Key and Value matrices project those vectors into Q, K and V. Their contiguous columns are split into the selected number of heads. Within each head, a query-key dot product is divided by the square root of head width, masked, and normalized across allowed keys using stable softmax. Each output coordinate sums probability times value. The head results are concatenated in head order and multiplied by the Output matrix. These are calculated arrays, not preassigned attention patterns or names suggesting learned syntax.

Use every matrix and head control

All five matrix editors accept rectangular JSON arrays with one through eight rows and columns, containing finite numbers from minus one hundred through one hundred. Input width is d; every projection must be d by d. Heads ranges from one through eight and must divide d exactly, with equal query, key and value width per head. A mismatched choice clears results and explains the constraint. Editing any matrix recomputes immediately. Inspect head selects which intermediate tables you see; it does not disable other heads or change the final output. Output tables always include every feature. Expanded Complete state evidence retains all heads at full numerical precision.

Separate masks, scaling and residuals

Causal mask allows a query to use its own key and earlier keys only. Key availability is a JSON array with one boolean per token; false excludes that token as a key for every query. It does not remove the token as a query. The two masks combine by logical AND. A query with no allowed key makes normalization undefined, so this inspector rejects the calculation rather than inventing uniform weights. Divide scores by square root of head width is the standard scaling; disabling it is an explicit unscaled comparison. With width one per head, that switch cannot change the answer because the divisor is already one. Add input residual adds Input after output projection without normalization.

Reproduce the default calculation

Reset defaults loads three tokens [1,0], [0,1], [1,1], identity projections, two heads, causal masking, scaling and no residual. Each head has one coordinate. Query one can use only key one, so its output is exactly [1,0]. For query two, head one has equal zero scores for its two allowed keys and returns one half. Head two has scores zero and one, giving output e divided by (1+e), approximately 0.731059. Query three returns approximately [0.844638,0.844638]. Inspect head two and verify its blocked future pairs carry zero probability even if the raw score is nonzero. Add the residual: the first output becomes [2,0] and the second becomes [0.5,1.731059].

Run a uniform and a masking experiment

Example only changes which fixture Load example will copy. Load example replaces all matrices, key availability and head count while preserving the mask, scaling and residual switches. Load Uniform queries and disable the residual. Zero queries make all allowed scores equal even though keys and values differ. With causal masking, outputs are [1,0], [0.5,0.5] and [0.666667,0.666667]. Turn the causal mask off and all three outputs become the mean of all three values. Change Key availability to [true,false,true]: every query now averages the first and third value, producing [1,0.5]. Use [false,false,false] to see the empty-key failure. Reset defaults restores valid inputs and head one inspection.

Distinguish splitting from averaging heads

At defaults select one head: both coordinates now participate in the same dot product and the divisor becomes square root of two. Query two assigns approximately 0.330238 and 0.669762 to keys one and two. Disable scaling and those weights become approximately 0.268941 and 0.731059. This differs from using two separate one-coordinate distributions. Re-enable scaling, then load Mixed projections. Q, K and V no longer equal Input, and the Output matrix sums and subtracts concatenated head coordinates. The first causal token has concatenated result [2,0] and projected result [2,2]. Inspect Output-projection terms to verify every multiply before its sum; averaging heads would not produce this result.

Interpret numerical evidence carefully

The projection table shows Q, K, V and the weighted result for the selected head. Pair scores are displayed before masking alongside allowed status and final weights; masked weights are exactly zero. Value-contribution rows expose each weighted term and output-projection rows expose mixing between heads. Tables scroll horizontally and can receive keyboard focus on narrow screens. This is a direct matrix inspector with immediate recalculation, so it needs no artificial play timer. JavaScript uses double precision; small probabilities can underflow to zero and displayed numbers are rounded. Stable softmax subtracts the largest allowed score before exponentiation. The bounded fixtures are hand-entered weights, not trained parameters or word embeddings. There is no tokenizer, positional encoding, dropout, bias, layer normalization, training, cross-attention, grouped-query attention, model-quality score or hardware benchmark.

Primary reference: Attention Is All You Need, sections 3.2.1 and 3.2.2. This bounded self-attention exercise uses square combined projections and equal per-head dimensions.