CPU Memory Budget for SLM Training

Why RAM is the binding constraint on a CPU

On a GPU, the memory wall and the compute wall are both close, and you feel them together. On a CPU running an SLM the two decouple sharply: you have relatively little compute but, on a typical machine, a fair amount of RAM (8–64 GB) and no separate device memory to overflow into. The result is that memory becomes a hard yes/no gate. Either the working set fits in physical RAM — and the model runs at whatever speed your cores and memory bandwidth allow — or it does not, and the operating system starts paging the model’s weights and cache to disk.

That second regime is catastrophic, not merely slow. Weights are read on every single token; if even a fraction of them live on an SSD instead of in RAM, each token pays disk latency thousands of times, and throughput collapses from tokens per second to seconds per token. This is why the memory budget is the first calculation you do, not the last. You are not optimizing a number here; you are checking a feasibility constraint. Everything else — quantization choice, context length, how many requests you batch — is downstream of the single inequality total_bytes ≤ usable_RAM. Get that wrong and no amount of kernel tuning saves you.

The accounting identity: five consumers of RAM

The entire budget is a sum of five terms. Writing them out as an identity makes the rest of the article a matter of expanding each one:

M_total  =  M_weights      (params × bytes/param)
          + M_kv           (2 × L × n_kv × d_head × N × B × p)
          + M_act          (activations / scratch, peak of prefill vs decode)
          + M_vocab        (tokenizer tables + logits buffer)
          + M_overhead     (runtime, allocator, fragmentation)

constraint:   M_total  ≤  usable_RAM  (physical RAM − OS − other apps)

Two of these terms are fixed once you pick a model and a precision: the weights and the vocab tables do not change as you run. The other three are dynamic: the KV cache grows linearly with context length N and batch B, activations peak during prefill, and overhead drifts up with fragmentation over a long-lived process. The art of budgeting is separating the fixed floor from the variable ceiling — the weights set the floor you can never go below, and everything you have left over above that floor is what you get to spend on context and concurrency. Keep this identity in view; every later section is just one term of it examined closely, and the worked example at the end is nothing but this sum evaluated with real numbers.

Model weights: params times bytes-per-param

The weight term is the simplest and usually the largest. A model with P parameters stored at b bytes each occupies M_weights = P × b bytes. That is the whole formula; the only subtlety is what b is, and that is set entirely by the numeric precision you load. The parameter count already bakes in every matrix in the model — the token embedding, each layer’s attention projections (W_Q, W_K, W_V, W_O) and its feed-forward matrices, the final norm and the output head — so you do not enumerate them separately for the budget; the published parameter count is the sum.

Concretely, take our running example: a 3.2B-parameter SLM (28 layers, d_model = 3072, 24 query heads of dimension 128, an 8192-wide SwiGLU feed-forward, vocabulary 128,000). At full 32-bit precision it needs 3.2e9 × 4 ≈ 12.8 GB — already too big for a typical 16 GB laptop once you add everything else. Halve the precision to 16-bit and it is 6.4 GB. The weight term dominates the floor, which is exactly why quantization — shrinking b — is the single highest-leverage lever in the entire budget. Everything else is measured in hundreds of megabytes; the weights are measured in gigabytes.

Precision and the bytes-per-param table

Because b is a single multiplier on the largest term, it is worth seeing every common choice side by side. The following table applies the P × b formula to our 3.2B model. Note the two unit conventions: GB here means 10^9 bytes (how model cards quote sizes) and GiB means 2^30 bytes (how your OS reports free RAM); they differ by about 7%, and mixing them is a classic way to be surprised by an out-of-memory error.

The jump from FP32 to a 4-bit format is an 8x reduction — from 12.8 GB down to under 2 GB. That is the difference between ‘does not fit on the laptop’ and ‘fits with room for a long context.’ The last row is important: real 4-bit formats are never exactly 4 bits per weight, because they store per-block scale and zero-point metadata alongside the packed integers, landing around 4.5 effective bits. Budget the real figure, not the idealized one.

Quantization: shrinking the weight term without breaking the model

Quantization replaces high-precision floats with low-bit integers plus a small amount of scaling metadata, so the weight term scales directly with the bit width. The reason it works at all is that neural-network weights are highly redundant and tolerant of rounding noise: the forward pass is a long chain of sums, and small per-weight errors average out rather than compounding. The reason it does not simply work at any bit width is that a naive global scale wastes range on outliers, so modern schemes quantize in small blocks (e.g. 32 or 64 weights share one scale), and mixed schemes keep the most sensitive tensors — attention W_V/W_O, the output head — at higher precision.

For the budget, the practical takeaways are concrete. INT8 is close to free in quality and halves the weights; it is the safe default when memory is merely tight. Four-bit k-quants (Q4_K_M and friends) quarter the FP16 size for a small, usually acceptable perplexity increase, and are what actually lets a 3B–7B model run comfortably on 8–16 GB machines. Below four bits (Q3, Q2) the quality loss grows fast and is rarely worth it for an already-small model. The rule of thumb: quantize the weights as aggressively as your quality bar tolerates, because every bit you shave off b multiplies against billions of parameters — nowhere else in the budget do you get that leverage.

The KV cache: deriving the formula

The second big term is the key-value cache. During autoregressive decoding, each new token must attend to every previous token, and recomputing the keys and values for the whole history at every step would be quadratic waste. Instead the model caches, for every past position, the key and value vectors it produced — so each step computes only the one new token’s K and V and appends them. That cache is pure memory spent to save compute, and its size follows directly from what it stores.

Count the elements. For each layer (there are L) and each KV head (there are n_kv, each of width d_head), you store one key vector and one value vector (that is the factor of 2) per token position (N) per sequence in the batch (B), at p bytes each:

M_kv  =  2 × L × n_kv × d_head × N × B × p

with d_kv = n_kv × d_head :   M_kv = 2 × L × d_kv × N × B × p

The cleanest way to hold this in your head is bytes per token, per sequence: 2 × L × d_kv × p. Everything about context length and batch is then just multiplication. For our model with grouped-query attention — 8 KV heads, so d_kv = 8 × 128 = 1024 — at FP16 that is 2 × 28 × 1024 × 2 = 114,688 bytes, or 112 KiB per token.

How context and batch grow the KV cache

The KV formula is linear in both N and B, and that linearity is the whole story of dynamic memory. At 112 KiB per token, our model’s cache grows in a way you can compute in your head: 1K tokens costs about 112 MiB, 8K tokens costs 8192 × 114,688 ≈ 0.94 GB (0.875 GiB), 32K tokens costs 3.5 GiB, and 128K tokens would cost 14 GiB — more than the weights themselves and more than a 16 GB laptop has. Long context is not free RAM; it is a term that can quietly overtake the model.

Batching multiplies the same per-token cost by the number of concurrent sequences, because every sequence keeps its own independent history. Serving 8 conversations at 8K context each costs 8 × 0.875 = 7 GiB of cache — the batch dimension is every bit as expensive as the context dimension. This is why, on memory-constrained CPUs, the KV cache and not the weights is usually what caps your concurrency: the weights are paid once and shared across all sequences in a batch, but the cache is paid per sequence. Understanding that N and B enter multiplicatively (N × B total cached tokens) is the key to sizing: what you actually have a budget for is a pool of cached tokens, and you spend it on length or on width.

GQA and MQA: shrinking the KV term at the source

The one architectural lever that attacks the KV cache directly is grouped-query attention. Standard multi-head attention (MHA) gives every query head its own key and value heads, so n_kv = n_heads. Grouped-query attention (GQA) lets several query heads share one KV head, and multi-query attention (MQA) is the extreme where all query heads share a single KV head (n_kv = 1). Since the cache size is proportional to n_kv — not to the number of query heads — this cuts the KV term by exactly the grouping ratio, while leaving the weights and the model’s expressivity almost untouched.

Our example makes the saving concrete. The model has 24 query heads; with full MHA the cache would use n_kv = 24, costing 2 × 28 × 24 × 128 × 2 = 344,064 bytes per token — a 2.625 GiB cache at 8K context. With 8 KV heads (a grouping ratio of 3) it is the 112 KiB/token and 0.875 GiB we computed — exactly one third. That 3x reduction is why essentially every modern SLM ships with GQA: it is a near-free 2–8x cut to the fastest-growing term in the budget, and it compounds with everything else. GQA and a 4-bit weight format together are what make long-context inference on a laptop possible at all.

KV cache quantization

The p in the KV formula is a lever too. Nothing requires the cache to be stored at the same precision as the compute; you can keep and reload keys and values at 8 bits, halving M_kv, and many runtimes now support this. Because the cache is read back and dotted against the current query, it is somewhat more sensitive to rounding than weights are — the errors feed straight into the attention scores — but 8-bit KV is widely usable with negligible quality impact, and 4-bit KV is viable for many workloads if you keep a few recent tokens at higher precision.

In budget terms, quantizing the cache to INT8 turns our 112 KiB/token into 56 KiB/token, so the 8K cache drops from 0.875 GiB to about 0.44 GiB, and the amount of context you can afford roughly doubles. Stack the three KV levers — GQA (cuts n_kv), context discipline (caps N × B), and cache quantization (cuts p) — and you control the dynamic memory term across more than an order of magnitude. That headroom is precisely what you trade back for longer context or more concurrent requests when you size the workload.

Activations and scratch: prefill versus decode

Activations are the intermediate tensors a forward pass produces — the residual stream, the attention scores, the feed-forward hidden state — and unlike training, inference does not retain them for a backward pass, so it only needs enough scratch for the layers currently in flight. That makes the activation term small, but it behaves very differently in the two phases of inference. During decode you process one token at a time, so the activation buffers are a handful of vectors of width d_model — kilobytes, effectively negligible.

During prefill you process the whole prompt of N tokens at once, so the intermediate buffers scale as B × N × d (residual) and B × N × d_ffn (feed-forward hidden), which for an 8K prompt is hundreds of megabytes if materialized whole. Two things keep this bounded: FlashAttention-style kernels never materialize the N × N score matrix, and chunked prefill processes the prompt in fixed-size windows so peak activation memory depends on the chunk size, not the full prompt length. Budget the activation term as the peak of the two phases — almost always prefill — and count on a good runtime keeping it to a few hundred megabytes rather than letting it scale with the full context.

The tokenizer, vocabulary, and logits buffer

The vocabulary shows up in the budget in three distinct places, and it helps to keep them separate. First, the embedding table and the output projection — both of size vocab × d_model — are large, but they are already counted inside P. For our model that is 128,000 × 3072 ≈ 393M parameters each (often tied, so counted once): a real chunk of the weight term, but not a separate line item. Do not double-count them.

Second, the tokenizer tables themselves — the merge rules and the token-to-id vocabulary a BPE or unigram tokenizer loads — are genuinely separate but tiny: a few megabytes of strings and rank tables, immaterial to the budget. Third, and easy to forget, is the logits buffer: the output head produces a distribution over the whole vocabulary, so it writes B × (tokens scored) × vocab floats. During decode you score only the last token, so that is 128,000 × 4 ≈ 0.5 MB — trivial. But if a runtime naively computes logits for every prompt position during prefill (B × N × vocab), an 8K prompt balloons to several gigabytes of logits. Good runtimes score only the needed positions; a bad configuration here is a surprisingly common OOM.

Framework overhead, mmap, and fragmentation

The last term is everything the accounting above ignores: the runtime’s own code and buffers, the memory allocator’s bookkeeping and slack, thread stacks, temporary tensors during model load, and fragmentation that accumulates in a long-lived process. Individually small, together they routinely add several hundred megabytes to a gigabyte, and they are real — budgeting to the exact byte of your theoretical sum and then being surprised by an OOM is a rite of passage. Reserve headroom of roughly 0.5–1 GB for it.

One CPU-specific detail cuts the other way and is worth understanding. Most inference runtimes mmap the weight file rather than reading it into a private allocation. The weights then live in the OS page cache, shared and backed by the file on disk, and are counted as clean, reclaimable pages. This means two processes running the same model can share one physical copy of the weights, and it means startup is nearly instant because pages fault in on demand. For budgeting, the weights still occupy their P × b bytes of physical RAM while in use — you cannot pretend mmap makes them free — but it does explain why resident-memory numbers in a process monitor can look lower or be shared, and why leaving a little RAM for the OS to keep those pages hot matters for steady-state speed.

The worked budget: 3.2B model, 8K context, 16 GB laptop

Now assemble the whole identity with real numbers. The machine is a 16 GB laptop; after the OS and a browser, assume about 13–14 GiB is usable. The model is our 3.2B SLM with GQA (8 KV heads), run at 8K context, batch 1. Two configurations are shown: a comfortable BF16 build, and an aggressively quantized build for an 8 GB machine.

The BF16 build lands around 8 GiB — it fits a 16 GB laptop with real headroom to spare for a longer prompt. The 4-bit build lands near 3.3 GiB, comfortably inside an 8 GB machine and leaving room to grow context. Notice the shape: weights dominate the fixed floor in both, while the cache is what will move as you change the workload. Quantization did the heavy lifting on the floor; GQA and KV quantization tamed the variable term.

Sizing context and batch to fit fixed RAM

Turn the budget inside out. The weights, overhead, and scratch are fixed once you choose a build, so subtract them from usable RAM and whatever remains is a token budget for the KV cache. Solve the KV formula for the total number of cached tokens:

KV_budget  =  usable_RAM − M_weights − M_act − M_overhead

max (N × B)  =  KV_budget  /  (2 × L × d_kv × p)
                  =  KV_budget  /  bytes_per_token

For the BF16 build on 14 GiB usable: after 5.96 GiB weights and ~1.2 GiB for scratch and overhead, roughly 6.84 GiB is left for cache. At 112 KiB per token that is about 64,000 cached tokens. You spend that pool however you like: one sequence at ~64K context, or eight concurrent sequences at 8K each, or any combination with N × B ≤ 64,000. This is the single most useful calculation in the whole article — it converts a fuzzy ‘will it fit?’ into a hard number. Switching to INT8 KV doubles the token budget to ~128K; switching to the 4-bit weight build frees another ~4 GiB of floor, roughly doubling it again. Every lever in this piece ultimately shows up as a bigger or smaller max (N × B), which is exactly the quantity you care about when you decide how long a document or how many users a machine can serve.

CPU realities: bandwidth, page cache, and NUMA

Fitting in RAM is necessary but not sufficient; on a CPU the next constraint is memory bandwidth, because decode is memory-bound. Every generated token streams the entire set of weights (and the growing KV cache) from RAM through the cores once. So a rough ceiling on decode speed is bandwidth / bytes_read_per_token: a laptop with ~50 GB/s of usable bandwidth reading 1.68 GiB of 4-bit weights per token tops out near 50 / 1.8 ≈ 28 tokens per second before compute even enters. This is the other reason quantization matters — smaller weights are not just easier to fit, they are faster to stream, so shrinking b speeds decode roughly proportionally.

Two more CPU details shape the budget in practice. Leave the OS enough free RAM to keep the mmap’d weight pages hot in the page cache; if you consume essentially all of RAM with cache and activations, the OS may evict weight pages and you re-fault them from disk mid-generation. And on multi-socket or big.LITTLE machines, NUMA effects mean memory has locality: weights allocated on one node but read by cores on another pay a cross-node penalty. Pinning threads and interleaving or localizing the model’s memory keeps effective bandwidth high. The budget tells you whether the model fits; bandwidth and locality tell you how fast it will then run.

Common pitfalls in memory budgeting

A handful of mistakes recur often enough to name. GB versus GiB: a ‘7 GB’ model file is ~6.5 GiB, and free-RAM tools report GiB — the ~7% gap has caused many a just-barely OOM. Forgetting the KV cache scales with context: a build that fits at 2K can OOM at 32K purely from cache growth, with the weights unchanged. Ignoring the batch multiplier: concurrency multiplies the cache, so a server that serves one user fine falls over at eight. Double-counting the embedding: the vocab × d tables are inside the parameter count already; adding them again inflates your estimate.

And the subtle ones. Prefill logits: a runtime that computes logits over all prompt positions can allocate gigabytes for nothing on a long prompt — score only the tokens you sample. Budgeting to zero headroom: allocator slack, fragmentation, and the OS need their 0.5–1 GB; a theoretical fit with no margin is a practical OOM. Confusing resident and shared memory: with mmap, a process monitor may under- or over-attribute the shared weight pages, so trust the formula over the number in the task manager. Every one of these is the same lesson: write the five-term sum down explicitly, in one unit, with headroom, and re-evaluate it whenever context or batch changes — the budget is a living constraint, not a one-time check.