Why architecture matters here
Quantization architecture matters because the quality-cost trade-offs are subtle and workload-specific. INT4 gives 4x memory reduction but 1-3 points of quality loss on complex tasks; FP8 gives 2x reduction with almost no loss; SmoothQuant preserves activation-outlier sensitivity that naive quantization breaks. Choosing wisely and calibrating well is the difference between a shipped model users love and one they complain about.
Cost is the fundamental driver. A production LLM serving cluster spends 60-80% of its budget on GPU. Cutting memory footprint by 4x means running 4x more concurrent users per GPU or using cheaper GPUs entirely. The economics of LLM serving depend on quantization.
Reliability matters because bad quantization silently degrades quality. Users don't complain about a single answer being slightly worse; they gradually lose trust as the model feels dumber. Continuous quality monitoring across the quantization boundary is what preserves user trust.
The architecture: every stage explained
Walk the diagram top to bottom.
FP16/BF16 Model. The base model as trained. FP16 for older hardware; BF16 for modern (A100+). Both are 2 bytes per weight. Sizes: 7B ≈ 14 GB, 70B ≈ 140 GB.
Calibration Data. A small representative dataset (128-512 examples) used to estimate activation ranges. Modern methods (GPTQ, AWQ) also use calibration to guide weight quantization. Domain-appropriate calibration produces better results than generic.
Quantization Method. GPTQ minimizes quantization error layer by layer using second-order information. AWQ scales down important weights before quantization to preserve their precision. SmoothQuant migrates the difficulty from activations to weights, letting both quantize cleanly. FP8 (E4M3 or E5M2) preserves dynamic range and works especially well on Hopper GPUs.
Weight Quantization. The largest cost saving. INT4 groupwise (128 weights share a scale, and one zero point) is the common format. Weights are packed 8-per-byte with scales stored separately. Memory drops 4x compared to FP16.
Activation Quant and KV Quant. Activations are quantized online during inference — computed statistics per token. INT8 is common; FP8 is emerging. KV cache quantization to INT8 or INT4 halves or quarters cache memory, crucial for long-context serving.
Fused Kernels. Dequantization is fused into the matmul kernel so the FP16 activations are computed against dequantized weights on the fly. This avoids materializing the full FP16 weights in HBM. Kernels are specific to the format; performance depends on quality of implementation.
Memory Layout. Weights, scales, and zero points must be laid out for kernel efficiency. Packed formats with scale interleaving are common. Layout mistakes tank throughput even if the math is correct.
Runtime. vLLM, TensorRT-LLM, llama.cpp, and MLC each ship with their own supported quantization formats and kernels. Choose based on hardware, model, and required feature set.
Quality Eval. MMLU, HumanEval, GSM8K, task-specific benchmarks. Compare against the FP16 baseline. Quality regression above 1-2 points is worth investigation.
Latency and Cost Eval. Tokens per second per GPU, HBM used, concurrent request capacity. These are the metrics that translate to cost per token.
Deployment. The quantized model gets pushed to HF Hub or the internal model registry. Serving config specifies quantization format and runtime. Rollout follows the same canary + monitoring pattern as any model change.
End-to-end quantization flow
Trace a quantization run. You want to quantize Llama-3-70B for INT4 serving on vLLM. You pick AWQ as the method.
You gather 128 calibration examples from your domain. You run AWQ: for each layer, it identifies the most important weight channels (those with high activation magnitudes) and applies a scaling that preserves their precision before quantization. Result: INT4 groupwise weights plus per-group scales, plus a small equivalence transform absorbed into the previous layer.
The quantized model is packed into a HuggingFace-compatible format. Total size: ~35 GB down from 140 GB. You upload to your model registry.
On the serving side, vLLM loads the model. Its AWQ kernel understands the packed weight format and the scales; it fuses dequantization with the matmul. HBM usage drops; you can now fit the model on a single H100 instead of two.
You run evals. MMLU: 78.5 vs 79.1 baseline — 0.6 points, acceptable. HumanEval: 71 vs 72. GSM8K: 84 vs 86. Domain-specific eval: 89 vs 90. All within your quality bar.
Latency: 28 tokens per second up from 15 (single-GPU config with more room for KV cache and larger batches). Cost per token: down 40%.
Canary at 5%. Monitor quality metrics online. After a week with no regression, promote to 100%.