SLM ARITHMETIC · WHERE THE PARAMETERS LIVE
SLM Parameter Breakdown
Count every weight of a published small model from its config, split into embedding, attention, FFN and norms, and check it against a memory budget.
Your experiment
Start with Gemma 2 2B against a 4 GB budget. Read which component holds the parameters, then switch models, change the attention heads, untie the embeddings, remove layers and change the budget.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Computed data
Metrics
Counts use each model's config.json: q, k, v and o projections (with q, k and v biases for Qwen2.5), three FFN matrices for the gated activation, RMSNorm weights (four per layer in Gemma 2) and the embedding and output head. The totals reproduce the published parameter counts. Memory counts weights only, at 2 bytes each.
Counting from the config
Every weight of a decoder-only transformer follows from a few numbers in config.json. Per layer, the query projection is d x H x hd, the key and value projections 2 x d x KV x hd, the output projection H x hd x d, the gated FFN 3 x d x ff, plus RMSNorm weights; once per model come the embedding V x d, the final norm and, if untied, an output head of the same size. For Gemma 2 2B, the opening Published configuration, this gives 2,614,341,888 parameters, the published count, and switching it to Qwen2.5-0.5B gives 494,032,768. Each total needs 2 bytes per weight in bf16.
Where they live
In Gemma 2 2B the FFN holds 63.4% of the parameters, attention 14.1% and the embedding table 22.6%, because a 256,000-token vocabulary times 2,304 dimensions is 589,824,000 weights. Qwen2.5-0.5B spends 27.6% on its embedding; Phi-3-mini-4k, with 32,064 tokens, spends 2.6% on the embedding plus another 98,500,608 on its untied output head. The smaller the model, the more a large vocabulary dominates.
Attention variants and tying
Attention heads changes how many key and value heads each layer has. Gemma 2 2B uses 8 query heads and 4 KV heads; full multi-head attention raises the total to 2,737,025,280 and a single shared KV head lowers it to 2,522,329,344, so the attention variant moves the parameter count by only a few percent while it moves the KV cache far more. Untying the embeddings adds a second 589,824,000-weight matrix: 3,204,165,888 in total.
The memory budget
At 2 bytes per weight Gemma 2 2B needs 5.23 GB, so it does not fit the opening 4 GB Memory for bf16 weights: DOES NOT FIT. Using Add or remove layers to remove 8 of its 26 layers brings it to 1,991,414,016 parameters and 3.98 GB, which fits, at the cost of whatever those layers computed; an 8 GB budget fits the unchanged model. Weights are only part of inference memory: the KV cache, activations and runtime buffers come on top.
Reading the tool and its limits
The chart stacks the components against the budget, the lanes list the configuration, and the table gives each component's formula, per-layer count, total and share. All numbers are computed when an input changes. The tool counts weights only, ignores quantization and assumes every listed matrix is stored densely. Rotary position embeddings have no weights and add nothing to the count.