Attention masks: permitted query-key pairs

Matrix experiment

Read the mask as a permission matrix

Each row is a query token position and each column is a key token position. A one permits that query to use that key; a zero forbids it. The diagonal represents attending to the current position, which is allowed in all four modes here. The matrix is a Boolean visibility rule, not learned attention weights. Real attention combines these permissions with scores and a normalization step. Counting permitted pairs therefore measures connectivity in this fixture, not model quality or actual inference speed.

Control sequence and visibility

Mask type selects causal, bidirectional, prefix-LM or a causal sliding window. Sequence length determines the square matrix size. Prefix length matters only in prefix mode: queries inside that prefix can use every prefix key, while later queries can use the prefix and all suffix keys up to themselves. Window size matters only in sliding mode and counts the current token as one of the visible positions. Query position selects one row for the readable key list. If you shorten the sequence, prefix, window and query values are clamped to remain valid. Restore defaults returns to an eight-token causal example.

Reproduce exact counts

With eight tokens, causal mode permits thirty-six of sixty-four pairs: the rows contain one through eight allowed keys. Bidirectional mode permits all sixty-four. Select prefix mode with prefix length four; replacing the triangular four-token prefix with a full four-by-four block adds six pairs, giving forty-two. A prefix length of zero reduces to causal attention and a prefix covering the whole sequence reduces to bidirectional attention. Select sliding mode with window three: the row counts are one, two, then six rows of three, totaling twenty-one.

Inspect the future-token boundary

Select query position two, using zero-based positions. Causal mode permits keys zero, one and two and excludes keys three onward. Bidirectional mode permits those future positions, so using it indiscriminately for next-token training would leak future-token information. Prefix mode intentionally permits future keys only inside the configured prefix. Sliding window one permits just the diagonal, removing historical context. Window eight matches causal mode for an eight-token sequence. The highlighted row is also written as text; color is not the only way to inspect a cell.

Keep architectural assumptions explicit

The fixture has no padding mask, packed examples, cross-attention, global tokens, sparse kernels or attention sink tokens. It does not assign a single mask to every model in a named product family. Prefix visibility is an explicit rule here rather than a claim about a particular implementation of T5 or another architecture. This is a direct matrix measurement, so edits recompute immediately and no artificial animation is required. Equal pair counts can still describe different patterns; inspect the actual rows before inferring equivalence.