SLM ARCHITECTURES · SIDE BY SIDE
SLM Architecture Comparison
Put two published small-model configurations side by side and compute what their choices cost in KV cache at a context length you pick.
Your experiment
Start with Phi-3-mini-4k against Gemma 2 2B at 8,192 tokens. Compare head counts, head size, FFN ratio and vocabulary, then change the context length, the KV precision and both models.
Every input recomputes the result immediately; there is no animation because nothing here unfolds over time. An input outside its allowed range is rejected with a message and the previous valid result stays on screen.
Computed data
Metrics
Hyperparameters are copied from each config.json (Llama 3.2 and Gemma 2 from public mirrors of the gated repositories) and licences from the model cards, fetched on 2026-10-10. KV cache counts keys and values for one sequence; Gemma 2 uses a 4,096-token sliding window on alternate layers, so half of its layers stop growing at 4,096. The Phi-3-mini-4k config sets a 2,047-token sliding window that Hugging Face transformers applies as an attention mask; whether a runtime also trims the cache varies, so its full cache is counted. Activations and runtime buffers are omitted.
Two published configurations
Model A and Model B load hyperparameters copied from each model's config.json. The opening pair is Phi-3-mini-4k, 3,821,079,552 parameters with 32 layers of full multi-head attention, 32 heads of size 96 and a 32,064-token vocabulary, against Gemma 2 2B, 2,614,341,888 parameters with 26 layers, 8 query heads sharing 4 key and value heads of size 256, and a 256,000-token vocabulary. The table lines up every choice, including the licence from each model card.
What a token costs in KV cache
Each cached token stores a key and a value per layer and KV head: 2 x layers x KV heads x head size x bytes. Phi-3-mini-4k needs 393,216 bytes per token in bf16, Gemma 2 2B 106,496 at full attention. At 8,192 tokens Phi-3 needs 3,072 MiB for one sequence and Gemma 2 624 MiB, because Gemma 2 also limits half of its layers to a 4,096-token sliding window. Grouped-query attention is the main reason the difference is nearly five times. The Phi-3-mini-4k config also lists a 2,047-token sliding window on every layer; the table shows it, but its cache is counted in full because not every runtime trims it.
Trained context is a hard edge
At the opening Context length of 8,192 tokens Phi-3-mini-4k is past its 4,096 trained positions: BEYOND TRAINED CONTEXT. At 4,096 both models are within range and need 1,536 and 416 MiB. At 131,072 tokens both are beyond their limits; the KV numbers would be 49,152 and 6,864 MiB, but the models were never trained on such positions. Llama 3.2 1B lists 131,072 positions and Qwen2.5 32,768 in their configs.
Precision and other pairs
KV cache precision at 8 bits halves every cache figure: 1,536 and 312 MiB at 8,192 tokens. Compare Llama 3.2 1B with Qwen2.5-0.5B at 32,768 tokens: 1,024 against 384 MiB, both within context. Reading the table, Gemma 2 runs a wide FFN, four times d, while Phi-3 uses 2.67 times d; Qwen2.5 keeps biases on its query, key and value projections.
Reading the tool and its limits
The bars compare one sequence's KV cache, the lanes show trained context and attention type, and the table lists sixteen properties per model. The cache counts keys and values only; it ignores paging overhead, batching, activations and the weights themselves. A config states what the architecture accepts, not how well a model performs, so quality comparisons need evaluations, not this table.