Why architecture matters here
The economics start with memory bandwidth. LLM inference at low batch sizes is bandwidth-bound: every generated token requires streaming essentially all model weights (and the growing KV cache) through the GPU's memory system, so tokens per second tracks bytes moved, almost linearly. Halving average bits per weight nearly halves per-token latency and doubles the model size that fits on a card. This is why quantization is not optional at serving scale — and why the average matters more than the minimum: a plan that holds 10% of weights at FP16 and 90% at INT4 moves barely more bytes than uniform INT4, while avoiding most of its quality loss.
Sensitivity is wildly non-uniform across a transformer, which is the empirical fact the whole approach rests on. Embeddings and the output head interact with the token distribution directly and degrade visibly when crushed. Early layers feed errors into everything downstream. Attention projections in some layers develop outlier channels — a few dimensions with magnitudes hundreds of times their neighbors, a well-documented emergent property of large models — that make naive per-tensor INT8 catastrophic while leaving neighboring layers perfectly tolerant. Meanwhile the wide FFN matrices, most of the parameter count, are typically the most forgiving. A uniform bit-width is thus dominated by its worst layer; mixed precision spends bits exactly where the model demands them.
The hardware supplies the third argument. Modern accelerators are not dtype-agnostic: tensor cores deliver specific accelerated paths for FP16, BF16, FP8, and INT8, with FP8 on recent generations doubling FP16 throughput. A precision plan is therefore also a hardware-mapping decision — the planner is choosing which silicon units each layer runs on — and a plan written for one GPU generation is not automatically right for the next. Treating precision as a per-layer, per-hardware, measured decision rather than a model-wide constant is what separates the serving stacks that get the promised speedups from those that get the promised accuracy loss with none of the speed.
The architecture: every piece explained
The calibration pipeline comes first because everything downstream consumes its output. It runs a few hundred to a few thousand representative sequences — sampled from real (redacted) traffic, not a generic corpus, because activation statistics are workload-dependent — through the full-precision model, recording per-layer activation ranges, outlier-channel maps, and per-channel weight statistics. The sensitivity profiler then measures what each layer can tolerate: quantize one layer at a time to each candidate precision, measure output divergence (KL divergence on logits, or task-eval deltas for the thorough version), and produce a sensitivity table — layer by precision by quality cost. Hessian-based approximations can cheapen this loop, but the honest per-layer perturbation study is affordable offline and hard to argue with.
The precision planner solves the allocation: given the sensitivity table, a memory budget (fit on an 80 GB card with room for KV cache at the target context length), a latency target, and a quality floor, assign each layer the cheapest precision whose cumulative predicted quality cost stays under the floor. In practice this is a greedy knapsack — demote the least-sensitive layers first, stop before the floor — refined by a final joint evaluation, since per-layer errors do not compose perfectly additively. The output is the precision manifest: a versioned artifact listing every tensor's dtype, quantization granularity (per-tensor, per-channel, or group-wise scales), and calibration constants. The manifest travels with the weights, is diffable between versions, and is the unit of rollback — deploying weights without their manifest, or vice versa, must be structurally impossible.
The runtime half makes the plan real. The quantizer packs weights per the manifest — group-wise INT4 with FP16 scales, per-channel INT8, FP8 with per-tensor scaling — producing the serving artifact. The kernel registry is the unglamorous linchpin: for every (weight dtype, activation dtype, hardware) combination the plan uses, there must be a fused kernel — dequantization folded into the GEMM's inner loop or epilogue, never a separate dequantize-to-FP16-then-multiply pass that doubles memory traffic and erases the win. The runtime dispatcher selects kernels per layer at model load and fails loudly — refusing to start, not silently falling back — if a required kernel is missing. Alongside, the KV-cache policy sets cache precision independently (FP8 with per-head scales is the current sweet spot), because at long context the cache, not the weights, dominates memory, and its precision trades against maximum batch size and context length. Quality gates — real task evals per plan, not perplexity alone — sit between planner and fleet.
End-to-end flow
A new fine-tuned checkpoint lands and needs to serve on the existing A100 fleet. The calibration pipeline samples two thousand production-shaped sequences and runs them through the FP16 checkpoint, logging activation ranges per layer; layer 14's attention output projection lights up with the familiar outlier channels — a few dimensions running two orders of magnitude hot. The profiler grinds through the perturbation study overnight: embeddings and head are sensitive as always, layers 1-2 and 14 are touchy, the FFN blocks from layer 8 onward barely notice INT4. The planner takes the table, an 80 GB budget with 32k-context KV headroom, and a quality floor of "no eval regresses more than 0.5%", and emits manifest v7: embeddings and head FP16, layers 1-2 and the outlier-prone projections FP8 with per-channel scales, everything else group-wise INT4 (group size 128), KV cache FP8 per-head. Predicted average: 4.9 bits per weight.
The quantizer packs weights to manifest v7; the dispatcher dry-loads it against the registry and confirms every layer resolves to a fused kernel on A100 — the INT4-weight-FP16-activation GEMM with fused group dequant, the FP8 paths, the FP8 KV attention kernel. Quality gates run the task suites: coding, extraction, long-context retrieval, safety refusals. Everything passes but long-context retrieval, down 1.1% — the joint effect of INT4 FFNs and FP8 cache compounding at depth, which per-layer prediction underestimated. The planner re-runs with the retrieval eval added to the floor; the revised manifest v7.1 promotes the last four FFN blocks to INT8, costing 0.2 bits of average width and recovering the eval. This iterate-on-evidence loop is the normal path, not the exception.
Deployment is atomic: weights plus manifest v7.1 roll to a canary slice of the fleet, and the runtime dashboards come alive — per-layer NaN/Inf counters at zero, activation ranges within calibration envelopes, latency down 38% versus FP16, tokens-per-second-per-GPU up 55%, quality canaries (a continuous trickle of golden prompts scored against reference answers) steady. After a clean soak, the fleet converges. Per request, nothing exotic happens — and that is the point: activations flow layer to layer in FP16/BF16, each layer's dispatcher-chosen kernel consumes quantized weights and fused scales, attention reads and writes FP8 cache entries, and the plan is invisible to everything except the memory bus and the invoice.