A transformer is, underneath, an enormous pile of matrix multiplications. The attention that lets a model relate one token to another, the feed-forward layers that do most of the thinking, the embedding table that turns words into vectors, and the final projection to logits over the vocabulary — every one of these is a matmul, or a short stack of them, with a nonlinearity sprinkled between. If you understand what a matrix multiplication is, what shapes it demands, what it costs, and why it is the operation hardware is built to do fast, you understand the load-bearing math of the entire architecture. This article builds that understanding from the ground up: vectors as the atoms, matrices as both containers and linear maps, the dot product and the geometry of projection, the learned weight matrices W_Q, W_K, W_V, and the rule — 2mnk — that lets you count the FLOPs in any matmul on the back of a napkin. We finish with why, for a small language model running on a CPU, matmul is not just a cost but the cost.
Why linear algebra is the language of transformers
Deep learning did not choose linear algebra for elegance; it chose it because linear algebra is the one kind of computation that modern hardware runs obscenely fast. A transformer’s job is to transform representations — take a sequence of vectors in and produce a sequence of vectors out — and the cheapest, most parallelizable way to transform a vector is to multiply it by a matrix. Stack billions of learned numbers into matrices, arrange the data as matrices too, and the forward pass becomes a chain of matrix multiplications interleaved with cheap element-wise functions.
This is why so much of ‘how a transformer works’ can be reduced to which matrices, of which shapes, multiply which tensors, in which order. The nonlinearities (softmax, GELU, layer norm) matter enormously for expressive power — without them the whole network would collapse into a single linear map — but they are computationally cheap. The heavy lifting, the FLOPs, the memory traffic, the time on the clock, all live in the matmuls. Learn to see a model as a graph of matrix multiplications and you can reason about its cost, its memory, and its bottlenecks without ever opening the training code.
Vectors: the atomic unit of meaning
The atom of a transformer is the vector: an ordered list of numbers, x = [x_1, x_2, …, x_d], living in a d-dimensional space. Each token — a word piece — is represented as one such vector of length d_model (the model width, often 512, 768, 4096, or larger). You should picture it geometrically: a point, or an arrow from the origin, in a high-dimensional space where direction encodes meaning. Tokens with similar meaning or role end up as vectors pointing in similar directions.
Two operations define a vector space and both appear constantly. Addition combines information: residual connections literally add a sublayer’s output vector back onto its input vector, so information accumulates rather than being overwritten. Scalar multiplication scales magnitude without changing direction, which is what scaling factors like 1/sqrt(d_k) in attention do. The length of a vector, ||x|| = sqrt(Σ_i x_i^2), is its magnitude; dividing by it gives a unit vector that keeps only the direction. Nearly every quantity a transformer manipulates — an embedding, a query, a hidden state, a gradient — is a vector in some space, and understanding what its dimensions and direction mean is half the battle.
Matrices: containers and linear maps at once
A matrix is a rectangular grid of numbers with shape [rows, cols], and it wears two hats. As a container, a matrix is just a stack of vectors: the activations for a sequence of N tokens, each of width d, form a matrix X: [N, d] — row i is token i’s vector. Data flows through the model as matrices (and, with a batch dimension, as 3-D tensors [B, N, d]).
The second, deeper hat is that a matrix is a linear map: a function that takes a vector in and returns a vector out, with the twin properties W(x + y) = Wx + Wy and W(αx) = α(Wx). A matrix W: [d_out, d_in] maps a d_in-vector to a d_out-vector, and every such linear map is a matrix — the two are the same thing. This is the conceptual key to transformers: the learned weight matrices are not passive lookup tables, they are learned linear transformations that rotate, scale, project, and mix the coordinates of an incoming vector into a new space where the next operation can do something useful. Training is the search for the entries of these maps.
The dot product: similarity and the seed of everything
The dot product of two vectors is the sum of their element-wise products: a · b = Σ_i a_i b_i. It returns a single scalar, and that scalar is a measure of alignment. Geometrically, a · b = ||a|| ||b|| cos(θ), where θ is the angle between the vectors. So the dot product is large and positive when two vectors point the same way, near zero when they are orthogonal (unrelated), and negative when they oppose. It is the numerical answer to ‘how similar are these two directions?’
This one scalar operation is the seed of the whole architecture. Attention scores a query against a key with a dot product: q · k asks ‘how relevant is this key to what I am looking for?’ A high dot product means high relevance, which after a softmax means high attention weight. And crucially, a matrix multiplication is nothing more than a grid of dot products — every entry of the output is the dot product of a row from one matrix with a column of the other. Master the dot product and matrix multiplication stops being mysterious: it is just many similarity computations packed together and run in parallel.
Matrix multiplication: the workhorse operation
Here is the operation the whole field is built on. To multiply A: [m, k] by B: [k, n] and get C: [m, n], each output entry is a dot product of a row of A with a column of B:
C[i, j] = Σ_p A[i, p] · B[p, j] for p = 1 … k
shapes: A: [m, k] B: [k, n] → C: [m, n]
^______^ the shared inner dimension k must matchThe rule that governs everything: the inner dimensions must agree. [m, k] × [k, n] works and yields [m, n]; the k is consumed. If the inner dimensions do not match, the multiplication is undefined — and the single most common bug in model code is a shape mismatch here. Matmul is not commutative (AB ≠ BA in general, and often one of the two is not even a legal shape), but it is associative: (AB)C = A(BC), which lets frameworks and hardware re-order and fuse chains of multiplications for efficiency. Every learned transformation in a transformer is an application of this rule; the art of reading a model is tracking the shapes through the chain.
Weight matrices as learned linear maps: W_Q, W_K, W_V
Now the payoff of ‘a matrix is a linear map.’ Self-attention begins by taking the input X: [N, d] and producing three different views of it — queries, keys, and values — by multiplying it against three learned weight matrices:
Q = X W_Q W_Q: [d, d_k] → Q: [N, d_k]
K = X W_K W_K: [d, d_k] → K: [N, d_k]
V = X W_V W_V: [d, d_v] → V: [N, d_v]Each of W_Q, W_K, W_V is a linear map that projects every token vector from the model space into a new, typically smaller, subspace. W_Q learns to extract ‘what this token is looking for’; W_K learns ‘what this token offers as a match’; W_V learns ‘what content to pass along if attended to.’ They are learned during training precisely because the useful projections are not obvious in advance — the network discovers which directions in representation space are worth comparing. These matrices are the bulk of what ‘the model knows’ about attention, and they are among the largest tensors in the network. Everything downstream is a consequence of the subspaces these three linear maps carve out.
Projection: the geometry of what those maps do
It is worth slowing down on the word projection, because it is the geometric intuition behind the weight matrices. To project a vector onto a direction u (a unit vector) is to ask ‘how much of this vector lies along u?’ — and the answer is a dot product: proj = (x · u) u. A matrix with several rows performs several such projections at once, reading out the components of x along several learned directions and assembling them into a new vector. That new vector is x re-expressed in a coordinate system the network finds useful.
So when W_Q maps a token from d = 768 dimensions down to d_k = 64, it is choosing 64 directions in the original space and recording how strongly the token points along each. Directions the task does not care about get small weights and effectively vanish; directions that carry signal get amplified. Dimensionality reduction, feature extraction, and ‘attention heads specializing in different relationships’ are all this same geometric act: a linear map projecting high-dimensional meaning onto a lower-dimensional subspace tuned for the comparison that comes next. Seeing weight matrices as learned projections, rather than opaque number grids, is what makes attention feel inevitable rather than arbitrary.
Shapes and broadcasting: the grammar you must obey
If matmul is the vocabulary, shape discipline is the grammar. Every tensor carries a shape, and operations impose rules on how those shapes combine. Matmul demands matching inner dimensions; element-wise operations (adding a bias, applying GELU, layer norm) demand matching shapes or a broadcastable pair. Broadcasting is the convention that lets a small tensor stretch to fit a larger one along size-1 axes: a bias vector b: [d] added to activations H: [N, d] is virtually copied across all N rows, so every token gets the same bias without materializing N copies.
Real models also carry a batch dimension, so activations are usually 3-D: [B, N, d] for B sequences of N tokens. Batched matmul applies the same weight matrix independently across the leading batch (and head) dimensions — [B, N, d] × [d, d_out] broadcasts the weight over all B sequences and yields [B, N, d_out]. Multi-head attention adds yet another axis, [B, heads, N, d_k], and the arithmetic runs in parallel across batch and heads. The mechanical skill that pays off forever is tracing shapes line by line: name every axis, check that inner dimensions meet at each matmul, and a large fraction of model bugs simply cannot occur.
Attention is matmuls all the way down
With queries, keys, and values in hand, the entire attention computation is two matmuls wrapped around a softmax:
scores = Q K^T / sqrt(d_k) [N, d_k] × [d_k, N] → [N, N]
A = softmax(scores) row-wise, shape stays [N, N]
out = A V [N, N] × [N, d_v] → [N, d_v]Read it as pure linear algebra. Q K^T is a grid of dot products: entry (i, j) is query i dotted with key j — the relevance of token j to token i. That is why the result is [N, N], one score per ordered pair, and why attention cost is quadratic in sequence length. The scaling by 1/sqrt(d_k) keeps the dot products from growing with dimension and pushing softmax into saturation. The softmax is the only non-linear step; it normalizes each row into attention weights that sum to one. Then A V is another matmul: each output row is a weighted sum of value vectors, the weights being that token’s attention distribution. Strip away the notation and attention is exactly three linear maps to build Q/K/V, one matmul to score, and one matmul to mix — matmuls bracketing a softmax.
The feed-forward network: the model’s biggest matmuls
After attention, each token passes independently through a feed-forward network (FFN, or MLP): two linear maps with a nonlinearity between them.
H = GELU(X W_1 + b_1) W_1: [d, d_ff] → [N, d_ff]
Y = H W_2 + b_2 W_2: [d_ff, d] → [N, d]The first matrix W_1 projects up to a wider hidden dimension — typically d_ff = 4d — giving the nonlinearity room to carve the space into many features; the second, W_2, projects back down to d. Because d_ff is several times d, these two matrices are usually the largest weight tensors in the model and the FFN typically accounts for roughly two thirds of the parameters and, at short-to-moderate context lengths, the majority of the FLOPs. It runs position-wise — the same W_1, W_2 applied identically to every token, with no interaction between positions — which is exactly what makes it a clean batched matmul. Attention moves information between tokens; the FFN does the heavy per-token computation. Both are, once again, matmuls.
Embeddings and logits: matmuls at the two ends
The pattern holds at the very entrance and exit of the model. The embedding layer turns token IDs into vectors via a table E: [V, d], where V is the vocabulary size. A one-hot token vector times E selects a row — so embedding lookup is formally a matrix multiplication by a one-hot, which is why it counts as linear algebra even though implementations shortcut it to a gather. At the far end, the final hidden state [N, d] is projected to a score for every vocabulary entry:
logits = H W_out W_out: [d, V] → [N, V]This output projection is a genuine, and often large, matmul — with a vocabulary of 50,000–150,000 tokens it can be one of the biggest single matmuls in a forward pass. Many models tie weights, reusing the transpose of the embedding table as W_out, saving parameters and coupling the ‘meaning of a token going in’ with ‘the score of predicting it coming out.’ A softmax over the logits produces the next-token distribution. From the first embedding to the last logit, both bookends are matmuls — the architecture is matmul-shaped end to end.
Counting the cost: 2mnk FLOPs for a matmul
Because matmul is the cost, you need to count it, and the rule is delightfully simple. To compute C = A B with A: [m, k] and B: [k, n], there are m × n output entries, and each is a dot product of length k — that is k multiplications and k - 1 additions, about 2k floating-point operations. So:
FLOPs(A[m,k] × B[k,n]) ≈ 2 · m · n · k
m · n output entries
× 2k FLOPs each (k mults + k adds)
= 2 m n kThat factor of two is the multiply-and-add pair, which is exactly why hardware exposes a fused multiply-add (FMA) and why chip vendors quote peak throughput in FMA-heavy terms. The formula scales with all three dimensions at once: double any of m, n, or k and you double the FLOPs. For a whole model you simply sum 2mnk over every matmul in the forward pass; a widely used shortcut is that a dense transformer costs about 2 × N × P FLOPs to process N tokens with P parameters, because each parameter participates in one multiply-add per token. Memorize 2mnk and you can size any layer.
A worked example: one attention projection and one FFN
Let us put real numbers in. Take d = 768, a batch of one sequence of N = 1024 tokens, d_ff = 3072 (that is 4d). First the query projection Q = X W_Q with X: [1024, 768] and W_Q: [768, 768]:
Q = X W_Q : m=1024, k=768, n=768
FLOPs = 2 · 1024 · 768 · 768 ≈ 1.21 × 10^9 (1.2 GFLOP)
FFN up-projection X W_1 : m=1024, k=768, n=3072
FLOPs = 2 · 1024 · 768 · 3072 ≈ 4.83 × 10^9 (4.8 GFLOP)Two things jump out. First, the FFN matmul is about four times the query projection, purely because n went from 768 to 3072 — the 2mnk rule made that visible instantly. Second, these are the costs of a single matmul in a single layer; a real model has W_Q, W_K, W_V, the output projection, two FFN matmuls, the two attention-score matmuls, all repeated across dozens of layers. Summed up, a forward pass over 1024 tokens easily runs to tens or hundreds of GFLOP. Counting is not academic: it tells you directly how long a prompt will take and where the time goes — and it says the time goes into big dense matmuls, with the FFN leading.
Why almost everything reduces to a matmul
Step back and the pattern is total. Attention scores: matmul. Attention output: matmul. Q/K/V projections: matmuls. FFN: two matmuls. Embedding lookup: a matmul by a one-hot. Output logits: a matmul. Even the residual stream is just vector addition threaded between them. The non-matmul operations — softmax, GELU, layer norm, the scaling constant — are element-wise or row-wise, cheap in FLOPs, and exist mainly to inject the nonlinearity that keeps the stacked linear maps from collapsing into one. Take the nonlinearities out and a hundred-layer transformer would be algebraically equivalent to a single matrix, capable of nothing but a linear function of its input.
So the honest one-sentence summary of a transformer is: a long chain of matrix multiplications with cheap nonlinear glue between them. This is not a simplification that loses something important — it is the reason the architecture is trainable and fast. Gradients flow cleanly through linear maps; GPUs and modern CPUs are built to multiply matrices at enormous rates; and the whole ecosystem of quantization, kernel fusion, and accelerator design exists to make that one operation faster. Understand the matmul and you understand where the parameters, the memory, the FLOPs, and the wall-clock time all go.
CPU and SLM relevance: matmul is the bottleneck
For a small language model on a CPU, this stops being theory and becomes the entire performance story. A CPU has no thousands of GPU cores; it leans on a handful of cores, SIMD vector units (AVX2, AVX-512, NEON), and a cache hierarchy. The dominant cost of running an SLM — generating each token — is the sequence of matmuls against the weight matrices, and on a CPU that work is carried by an optimized GEMM (general matrix multiply) kernel. How well that kernel tiles the matrices to fit in L1/L2 cache, reuses loaded data, and saturates the vector units is, very nearly, the model’s speed.
Two facts shape CPU inference. During single-token decode, each matmul is a skinny [1, k] × [k, n] — a matrix-vector product that reads a huge weight matrix to do relatively little arithmetic, so it is memory-bandwidth-bound, limited by how fast weights stream from RAM. That is exactly why quantization (int8, int4) is the headline CPU optimization: shrinking each weight from 32 or 16 bits to 4 or 8 bits cuts the bytes moved per matmul several-fold and speeds up the memory-bound step almost proportionally, while also letting integer SIMD do more multiply-adds per instruction. Prompt prefill, by contrast, batches many tokens into fat matmuls that are compute-bound and love the vector units. Either way the lever is the same operation. If you want a CPU SLM to go faster, you make its matmuls cheaper — smaller weights, better GEMM kernels, cache-friendly shapes — because the matmul is the bottleneck, full stop.
Common pitfalls when reasoning in shapes and FLOPs
A few traps catch almost everyone. First, transpose confusion: Q K^T needs the transpose so the inner dimensions meet — [N, d_k] × [d_k, N] — and forgetting it is the classic shape-mismatch crash. Second, non-commutativity: AB and BA are different maps and usually not even both legal; order is meaning, not a detail. Third, double-counting or half-counting FLOPs: the factor of two in 2mnk is the multiply and the add — drop it and you undercount by half; count a fused multiply-add as two separate instructions and you may double-count on hardware that does it in one.
Fourth, ignoring the batch and head axes: a per-token matmul cost must be multiplied by B × heads × layers to get a real total, and it is easy to reason about one token and forget the model runs thousands. Fifth, conflating parameters with FLOPs: attention has relatively few parameters but its score matmuls grow as N^2, so at long context it can dominate compute even though the FFN dominates the parameter count. And finally, assuming FLOPs equal time: on a CPU a memory-bound matrix-vector decode step can be far slower per FLOP than a compute-bound prefill, because the limiter is bytes moved, not math done. Keep shapes named, keep the 2mnk honest, and remember whether you are compute-bound or memory-bound, and the linear algebra will not surprise you.
W_Q, W_K, W_V, the FFN weights, the embedding and output projections — that transform them. The dot product measures alignment, and a matmul is just a grid of dot products with the rule that inner dimensions must match. Attention, the feed-forward network, embeddings, and logits all reduce to matmuls, and you can count any of them with 2mnk FLOPs — the factor of two being the multiply-and-add pair hardware fuses. For a small language model on a CPU this is the whole game: matmul is the compute and memory bottleneck, decode is memory-bandwidth-bound, and the fastest way to speed the model up is to make its matmuls cheaper through quantization and cache-friendly GEMM kernels. Learn to see the shapes, learn to count the FLOPs, and the architecture stops being a black box.