Why linear algebra is the language of transformers

Deep learning did not choose linear algebra for elegance; it chose it because linear algebra is the one kind of computation that modern hardware runs obscenely fast. A transformer’s job is to transform representations — take a sequence of vectors in and produce a sequence of vectors out — and the cheapest, most parallelizable way to transform a vector is to multiply it by a matrix. Stack billions of learned numbers into matrices, arrange the data as matrices too, and the forward pass becomes a chain of matrix multiplications interleaved with cheap element-wise functions.

This is why so much of ‘how a transformer works’ can be reduced to which matrices, of which shapes, multiply which tensors, in which order. The nonlinearities (softmax, GELU, layer norm) matter enormously for expressive power — without them the whole network would collapse into a single linear map — but they are computationally cheap. The heavy lifting, the FLOPs, the memory traffic, the time on the clock, all live in the matmuls. Learn to see a model as a graph of matrix multiplications and you can reason about its cost, its memory, and its bottlenecks without ever opening the training code.

Advertisement

Vectors: the atomic unit of meaning

The atom of a transformer is the vector: an ordered list of numbers, x = [x_1, x_2, …, x_d], living in a d-dimensional space. Each token — a word piece — is represented as one such vector of length d_model (the model width, often 512, 768, 4096, or larger). You should picture it geometrically: a point, or an arrow from the origin, in a high-dimensional space where direction encodes meaning. Tokens with similar meaning or role end up as vectors pointing in similar directions.

Two operations define a vector space and both appear constantly. Addition combines information: residual connections literally add a sublayer’s output vector back onto its input vector, so information accumulates rather than being overwritten. Scalar multiplication scales magnitude without changing direction, which is what scaling factors like 1/sqrt(d_k) in attention do. The length of a vector, ||x|| = sqrt(Σ_i x_i^2), is its magnitude; dividing by it gives a unit vector that keeps only the direction. Nearly every quantity a transformer manipulates — an embedding, a query, a hidden state, a gradient — is a vector in some space, and understanding what its dimensions and direction mean is half the battle.

Advertisement

Matrices: containers and linear maps at once

A matrix is a rectangular grid of numbers with shape [rows, cols], and it wears two hats. As a container, a matrix is just a stack of vectors: the activations for a sequence of N tokens, each of width d, form a matrix X: [N, d] — row i is token i’s vector. Data flows through the model as matrices (and, with a batch dimension, as 3-D tensors [B, N, d]).

The second, deeper hat is that a matrix is a linear map: a function that takes a vector in and returns a vector out, with the twin properties W(x + y) = Wx + Wy and W(αx) = α(Wx). A matrix W: [d_out, d_in] maps a d_in-vector to a d_out-vector, and every such linear map is a matrix — the two are the same thing. This is the conceptual key to transformers: the learned weight matrices are not passive lookup tables, they are learned linear transformations that rotate, scale, project, and mix the coordinates of an incoming vector into a new space where the next operation can do something useful. Training is the search for the entries of these maps.

The dot product: similarity and the seed of everything

The dot product of two vectors is the sum of their element-wise products: a · b = Σ_i a_i b_i. It returns a single scalar, and that scalar is a measure of alignment. Geometrically, a · b = ||a|| ||b|| cos(θ), where θ is the angle between the vectors. So the dot product is large and positive when two vectors point the same way, near zero when they are orthogonal (unrelated), and negative when they oppose. It is the numerical answer to ‘how similar are these two directions?’

This one scalar operation is the seed of the whole architecture. Attention scores a query against a key with a dot product: q · k asks ‘how relevant is this key to what I am looking for?’ A high dot product means high relevance, which after a softmax means high attention weight. And crucially, a matrix multiplication is nothing more than a grid of dot products — every entry of the output is the dot product of a row from one matrix with a column of the other. Master the dot product and matrix multiplication stops being mysterious: it is just many similarity computations packed together and run in parallel.

Matrix multiplication: the workhorse operation

Here is the operation the whole field is built on. To multiply A: [m, k] by B: [k, n] and get C: [m, n], each output entry is a dot product of a row of A with a column of B:

C[i, j] = Σ_p  A[i, p] · B[p, j]      for p = 1 … k

shapes:   A: [m, k]   B: [k, n]   →   C: [m, n]
               ^______^  the shared inner dimension k must match

The rule that governs everything: the inner dimensions must agree. [m, k] × [k, n] works and yields [m, n]; the k is consumed. If the inner dimensions do not match, the multiplication is undefined — and the single most common bug in model code is a shape mismatch here. Matmul is not commutative (AB ≠ BA in general, and often one of the two is not even a legal shape), but it is associative: (AB)C = A(BC), which lets frameworks and hardware re-order and fuse chains of multiplications for efficiency. Every learned transformation in a transformer is an application of this rule; the art of reading a model is tracking the shapes through the chain.

Weight matrices as learned linear maps: W_Q, W_K, W_V

Now the payoff of ‘a matrix is a linear map.’ Self-attention begins by taking the input X: [N, d] and producing three different views of it — queries, keys, and values — by multiplying it against three learned weight matrices:

Q = X W_Q        W_Q: [d, d_k]   →   Q: [N, d_k]
K = X W_K        W_K: [d, d_k]   →   K: [N, d_k]
V = X W_V        W_V: [d, d_v]   →   V: [N, d_v]

Each of W_Q, W_K, W_V is a linear map that projects every token vector from the model space into a new, typically smaller, subspace. W_Q learns to extract ‘what this token is looking for’; W_K learns ‘what this token offers as a match’; W_V learns ‘what content to pass along if attended to.’ They are learned during training precisely because the useful projections are not obvious in advance — the network discovers which directions in representation space are worth comparing. These matrices are the bulk of what ‘the model knows’ about attention, and they are among the largest tensors in the network. Everything downstream is a consequence of the subspaces these three linear maps carve out.

Projection: the geometry of what those maps do

It is worth slowing down on the word projection, because it is the geometric intuition behind the weight matrices. To project a vector onto a direction u (a unit vector) is to ask ‘how much of this vector lies along u?’ — and the answer is a dot product: proj = (x · u) u. A matrix with several rows performs several such projections at once, reading out the components of x along several learned directions and assembling them into a new vector. That new vector is x re-expressed in a coordinate system the network finds useful.

So when W_Q maps a token from d = 768 dimensions down to d_k = 64, it is choosing 64 directions in the original space and recording how strongly the token points along each. Directions the task does not care about get small weights and effectively vanish; directions that carry signal get amplified. Dimensionality reduction, feature extraction, and ‘attention heads specializing in different relationships’ are all this same geometric act: a linear map projecting high-dimensional meaning onto a lower-dimensional subspace tuned for the comparison that comes next. Seeing weight matrices as learned projections, rather than opaque number grids, is what makes attention feel inevitable rather than arbitrary.

Shapes and broadcasting: the grammar you must obey

If matmul is the vocabulary, shape discipline is the grammar. Every tensor carries a shape, and operations impose rules on how those shapes combine. Matmul demands matching inner dimensions; element-wise operations (adding a bias, applying GELU, layer norm) demand matching shapes or a broadcastable pair. Broadcasting is the convention that lets a small tensor stretch to fit a larger one along size-1 axes: a bias vector b: [d] added to activations H: [N, d] is virtually copied across all N rows, so every token gets the same bias without materializing N copies.

Real models also carry a batch dimension, so activations are usually 3-D: [B, N, d] for B sequences of N tokens. Batched matmul applies the same weight matrix independently across the leading batch (and head) dimensions — [B, N, d] × [d, d_out] broadcasts the weight over all B sequences and yields [B, N, d_out]. Multi-head attention adds yet another axis, [B, heads, N, d_k], and the arithmetic runs in parallel across batch and heads. The mechanical skill that pays off forever is tracing shapes line by line: name every axis, check that inner dimensions meet at each matmul, and a large fraction of model bugs simply cannot occur.

Attention is matmuls all the way down

With queries, keys, and values in hand, the entire attention computation is two matmuls wrapped around a softmax:

scores = Q K^T / sqrt(d_k)     [N, d_k] × [d_k, N]  →  [N, N]
A      = softmax(scores)       row-wise, shape stays  [N, N]
out    = A V                   [N, N] × [N, d_v]    →  [N, d_v]

Read it as pure linear algebra. Q K^T is a grid of dot products: entry (i, j) is query i dotted with key j — the relevance of token j to token i. That is why the result is [N, N], one score per ordered pair, and why attention cost is quadratic in sequence length. The scaling by 1/sqrt(d_k) keeps the dot products from growing with dimension and pushing softmax into saturation. The softmax is the only non-linear step; it normalizes each row into attention weights that sum to one. Then A V is another matmul: each output row is a weighted sum of value vectors, the weights being that token’s attention distribution. Strip away the notation and attention is exactly three linear maps to build Q/K/V, one matmul to score, and one matmul to mix — matmuls bracketing a softmax.

The feed-forward network: the model’s biggest matmuls

After attention, each token passes independently through a feed-forward network (FFN, or MLP): two linear maps with a nonlinearity between them.

H = GELU(X W_1 + b_1)     W_1: [d, d_ff]     →  [N, d_ff]
Y = H W_2 + b_2          W_2: [d_ff, d]     →  [N, d]

The first matrix W_1 projects up to a wider hidden dimension — typically d_ff = 4d — giving the nonlinearity room to carve the space into many features; the second, W_2, projects back down to d. Because d_ff is several times d, these two matrices are usually the largest weight tensors in the model and the FFN typically accounts for roughly two thirds of the parameters and, at short-to-moderate context lengths, the majority of the FLOPs. It runs position-wise — the same W_1, W_2 applied identically to every token, with no interaction between positions — which is exactly what makes it a clean batched matmul. Attention moves information between tokens; the FFN does the heavy per-token computation. Both are, once again, matmuls.

Embeddings and logits: matmuls at the two ends

The pattern holds at the very entrance and exit of the model. The embedding layer turns token IDs into vectors via a table E: [V, d], where V is the vocabulary size. A one-hot token vector times E selects a row — so embedding lookup is formally a matrix multiplication by a one-hot, which is why it counts as linear algebra even though implementations shortcut it to a gather. At the far end, the final hidden state [N, d] is projected to a score for every vocabulary entry:

logits = H W_out       W_out: [d, V]   →   [N, V]

This output projection is a genuine, and often large, matmul — with a vocabulary of 50,000–150,000 tokens it can be one of the biggest single matmuls in a forward pass. Many models tie weights, reusing the transpose of the embedding table as W_out, saving parameters and coupling the ‘meaning of a token going in’ with ‘the score of predicting it coming out.’ A softmax over the logits produces the next-token distribution. From the first embedding to the last logit, both bookends are matmuls — the architecture is matmul-shaped end to end.

Counting the cost: 2mnk FLOPs for a matmul

Because matmul is the cost, you need to count it, and the rule is delightfully simple. To compute C = A B with A: [m, k] and B: [k, n], there are m × n output entries, and each is a dot product of length k — that is k multiplications and k - 1 additions, about 2k floating-point operations. So:

FLOPs(A[m,k] × B[k,n])  ≈  2 · m · n · k

  m · n  output entries
  × 2k  FLOPs each (k mults + k adds)
  = 2 m n k

That factor of two is the multiply-and-add pair, which is exactly why hardware exposes a fused multiply-add (FMA) and why chip vendors quote peak throughput in FMA-heavy terms. The formula scales with all three dimensions at once: double any of m, n, or k and you double the FLOPs. For a whole model you simply sum 2mnk over every matmul in the forward pass; a widely used shortcut is that a dense transformer costs about 2 × N × P FLOPs to process N tokens with P parameters, because each parameter participates in one multiply-add per token. Memorize 2mnk and you can size any layer.

A worked example: one attention projection and one FFN

Let us put real numbers in. Take d = 768, a batch of one sequence of N = 1024 tokens, d_ff = 3072 (that is 4d). First the query projection Q = X W_Q with X: [1024, 768] and W_Q: [768, 768]:

Q = X W_Q :  m=1024, k=768, n=768
  FLOPs = 2 · 1024 · 768 · 768  ≈  1.21 × 10^9  (1.2 GFLOP)

FFN up-projection  X W_1 :  m=1024, k=768, n=3072
  FLOPs = 2 · 1024 · 768 · 3072  ≈  4.83 × 10^9  (4.8 GFLOP)

Two things jump out. First, the FFN matmul is about four times the query projection, purely because n went from 768 to 3072 — the 2mnk rule made that visible instantly. Second, these are the costs of a single matmul in a single layer; a real model has W_Q, W_K, W_V, the output projection, two FFN matmuls, the two attention-score matmuls, all repeated across dozens of layers. Summed up, a forward pass over 1024 tokens easily runs to tens or hundreds of GFLOP. Counting is not academic: it tells you directly how long a prompt will take and where the time goes — and it says the time goes into big dense matmuls, with the FFN leading.