Loading is only the first allocation
A checkpoint loading successfully answers a narrow question: the runtime allocated the model under the conditions present at startup. Generating a response changes those conditions. The engine may allocate attention state, execution buffers and additional request slots. A model that loads can therefore fail later as the prompt grows or another request arrives. The correct capacity question includes the workload, not just the checkpoint size.
Start by separating persistent weights from request-dependent state. Lower-precision weights can release memory, but their compression ratio is not the compression ratio of the complete service. A sixteen-bit attention cache can remain sixteen-bit even when selected weights use four-bit storage. Some layers and metadata also retain different representations. A realistic budget names these components rather than treating the file size as the device allocation.
Why context and concurrency matter
For a simplified decoder with uniform layers, an uncompressed key/value cache has two tensors per layer. Multiply two by the layer count, key/value head count, head dimension, cached tokens, concurrent sequences and bytes per element. The result is an illustrative cache payload, before allocator, paging and runtime overhead. Use key/value heads, not query heads, for architectures that share keys and values across attention heads.
Longer contexts increase the number of cached tokens. Concurrency increases the number of active sequences. Their combined effect can dominate the memory released by weight compression. Prefix sharing, sliding-window attention and other architecture-specific behavior can change actual allocation; the simplified equation cannot infer those optimizations. Read the configuration of the actual model and verify the engine behavior before using a calculated number for a capacity commitment.
Lab: vary the request shape
This browser exercise uses a hypothetical uniform model with 24 layers, 8 key/value heads and a head dimension of 64. It assumes a sixteen-bit uncompressed cache. The weight allowance is 2 GiB, other runtime allowance is 1 GiB, and available memory is 8 GiB. Change tokens per sequence and simultaneous sequences, then inspect the calculated cache payload and remaining budget. None of these constants describe a measured checkpoint.
Try 2,048 tokens with one sequence, then double concurrency and context separately. Observe that both inputs scale the simplified cache estimate linearly. The expected lesson is the relationship between workload and memory, not whether a named model fits. The display reports a negative remainder as an exceeded illustrative budget. A positive remainder is only arithmetic headroom; it is not a guarantee of successful allocation.
Educational calculation on synthetic inputs. No device access, model calls or measured GPU results.
Measure a boundary rather than guessing
For an actual run, hold the model revision and precision constant. Begin with short prompts and one active request. Increase one input at a time, recording peak device memory, request success and latency. Keep output limits explicit because generated tokens also extend sequence state. Warmup policy matters: the first request may trigger initialization that later requests reuse.
Record failed runs alongside successful ones. If a context limit rejects a request before allocation, label it as a configuration limit rather than an out-of-memory observation. If a background application uses the same GPU, report that shared-device limitation. A device-wide memory sample includes other processes and must not be described as exact model memory. Allocator statistics and device samples answer different questions.
Choose a workload envelope
A practical outcome is a tested envelope: a specified model, context length, concurrency and output limit that completed a declared set of requests. Leave margin for variability rather than selecting the last successful allocation as a safe maximum. Repeat representative cases and report the number of trials. A single short success does not demonstrate sustained capacity.
This guide contains an explanatory simulation, not GPU benchmark results. Use the quantization guide to estimate weight storage, this exercise to understand request-dependent growth, and the measured-results page only when recorded experiments exist. Together they separate a useful mental model from device-specific evidence. All examples are synthetic, and no prompt text or device identifier is collected by this exercise.
Sources and next steps
vLLM quantization compatibility · PEFT LoRA explanation · QLoRA paper
Follow the Local LLM learning path. Evidence label: explanation and calculated simulation. All inputs are synthetic; no measured training or inference claim is made.