All 48 articles, sorted alphabetically
AWQ vs GPTQ Quantization, in depth: what each objective is sensitive to, a fair bake-off, and a decision rule
How to choose between AWQ and GPTQ for 4-bit weight-only LLM quantization: what each method optimises, how calibration shift, act-order, group size an…
Read article →bitsandbytes vs AutoAWQ vs AutoGPTQ, in depth: what each tool is in 2026, checkpoint formats, training versus serving, and a migration plan
A practical comparison of the three libraries most tutorials still name: what bitsandbytes, AutoAWQ and AutoGPTQ actually do, why AutoAWQ and AutoGPTQ…
Read article →FP8 Quantization for LLM Inference: E4M3 vs E5M2, Scaling Factors and Accuracy
Choose FP8 formats and scale recipes for LLM inference from evidence: E4M3 and E5M2 bit layouts, OCP versus FNUZ variants, scale arithmetic, per-tenso…
Read article →FP8 Inference Explained, in depth: E4M3 numerics, scale granularity, the scaled GEMM, FP8 KV cache, and shipping a checkpoint with vLLM
A practical guide to serving LLMs in FP8: what E4M3 can represent, how scales map tensors into it, per-tensor versus per-channel and per-token versus …
Read article →GGUF Format Explained, in depth: reading, validating and debugging a model file byte by byte
An engineer's guide to the GGUF container used by llama.cpp: the header, typed metadata, tensor infos and aligned data section, a complete Python…
Read article →GPU Kernels for INT4 Inference, in depth: writing a W4A16 kernel in Triton, testing it against the right reference, benchmarking it against the bandwidth roofline and knowing what production kernels add
A hands-on guide to INT4 weight-only GPU kernels: why decode is bandwidth-bound, the packed data contract, a working W4A16 GEMM in Triton explained li…
Read article →INT8 vs INT4 Quantization, in depth: memory-bound versus compute-bound serving, the crossover batch size and how to decide
How to choose between INT8 and INT4 quantization for a model you actually serve: what each format saves, why weight-only INT4 wins memory-bound decodi…
Read article →INT8 Calibration for LLMs, in depth: choosing static activation scales, observers, calibration data, sensitivity sweeps and the evaluation gate
How to calibrate an LLM for static INT8 (W8A8) inference: what a scale controls, why LLM activations defeat min-max, how to choose calibration data, o…
Read article →BF16 (Brain Float), in depth: the format as a storage, inference and quantization baseline
BF16 from the bits up for people who store, serve and quantize models: the 1-8-7 layout, range and precision against FP32, FP16 and FP8, round-to-near…
Read article →FP16 (Half Precision), in depth: the bit layout, range and rounding limits, loss scaling, overflow in inference and when to choose it over BF16
IEEE half precision from first principles: bit layout, exact limits, rounding and conversion, FP32 accumulation, dynamic loss scaling with PyTorch cod…
Read article →GPTQ, in depth: auditing a 4-bit checkpoint, from a one-layer lab to calibration data, Hessian diagnostics and release gates
A practical GPTQ audit guide: a one-layer GPTQ versus round-to-nearest lab in PyTorch, Hessian size and rank arithmetic for a 7B model, calibration da…
Read article →INT4 Quantization, in depth: the 4-bit format, group scales, error budgets and when to use it
INT4 quantization from the bits up: the 16-level integer grid, scale and zero point, group size and true bits per weight, a group quantized by hand, o…
Read article →INT8 Quantization, in depth: the integer arithmetic from scale and zero point to int32 accumulation, requantization and the kernels that run it
How INT8 inference actually computes: the affine mapping, the zero-point algebra inside an int8 GEMM, int32 accumulation and headroom, fixed-point req…
Read article →Quantization Overview, in depth: what to quantize, which number format, which method, which kernel, and how to prove it worked
A map of model quantization from first principles: the affine mapping with a worked example, why fewer bits speed up memory-bound inference, weights v…
Read article →SmoothQuant, in depth: choosing alpha per layer, measuring the error it removes, and the methods that grew out of it
A practical guide to tuning SmoothQuant on a real model: collecting per-channel activation statistics with forward hooks, sweeping the migration stren…
Read article →QLoRA Fine-Tuning Explained, in depth: the 4-bit training step, where the memory goes, and how to merge without losing what you trained
How QLoRA fine-tunes a large model on one GPU: a frozen NF4 base dequantized on the fly, LoRA adapters that carry all gradients and optimizer state, a…
Read article →Activation Quantization Architecture, in depth: quantize sites, integer GEMM arithmetic, fused kernels and per-site debugging
The activation side of quantized inference as a datapath: why activations are harder than weights, which tensors in a transformer block get quantized …
Read article →Adaptive rounding architecture
Deep-dive on adaptive rounding (AdaRound): why round-to-nearest is optimal for the wrong objective, layer-wise reconstruction on calibration activatio…
Read article →AWQ architecture, in depth: the calibration pipeline, scale search, packed format and serving kernels
How activation-aware weight quantization is built as a system: block-by-block calibration, the smoothing pairs where scales are folded, the scale sear…
Read article →AWQ vs GPTQ Quantization Architectures Compared: Hessian error compensation versus activation-aware scaling
How GPTQ and AWQ work inside: the layer objective, Hessian-based error compensation versus activation-aware channel scaling, code for both, checkpoint…
Read article →Bit Packing Architecture, in depth: sub-byte layouts, pack axes and slot orders, odd bit widths, effective bits per weight, repacking between formats and verification
How quantized weights are packed into bytes and words: the general packing arithmetic, containers, pack axis and slot order, the real GPTQ, AWQ, GGUF …
Read article →BitNet b1.58 and BitLinear: Ternary 1.58-Bit LLM Weights
How BitNet b1.58 works: the BitLinear layer, ternary {-1, 0, +1} weights with absmean scaling, int8 activations, straight-through training, add/subtra…
Read article →Quantization Calibration: Min/Max vs Percentile vs KL Divergence
How calibration picks INT8 quantization ranges: representative data, min/max vs percentile clipping, KL entropy and MSE methods, per-channel scales, P…
Read article →Double Quantization: Quantizing the Quantization Constants
What double quantization is: 4-bit block-wise weights carry one FP32 scale per block (about half a bit per weight), so the scales are quantized again …
Read article →Static vs Dynamic Quantization: Calibration and INT8 Trade-offs
Static vs dynamic quantization: static calibration bakes activation scales ahead of time; dynamic computes them at runtime. INT8 accuracy, latency, mi…
Read article →FP8 -- 8-bit floating point for training and inference
Deep-dive on FP8: the E4M3 and E5M2 formats, floating point vs integer (dynamic range vs uniform steps), scaling, native tensor-core support (Hopper/B…
Read article →GGUF Quantization: Q4_K_M, I-Quants, imatrix, GPU Offload
How GGUF quantization works in llama.cpp: per-block scales, Q4_K_M super-blocks, IQ2 and IQ3 with an importance matrix, mmap, GPU offload, choosing a …
Read article →GPTQ Architecture, in depth: the block-by-block pipeline, Cholesky error feedback, memory at scale, groups, act-order and packed INT4 serving
How GPTQ quantises LLM weights to 3-4 bits without retraining: the per-layer update in brief, the block-by-block pipeline, a working implementation, m…
Read article →Quantization granularity -- per-tensor, per-channel, per-group scales
Deep-dive on quantization granularity: per-tensor vs per-channel vs per-group scales, scale and zero-point, the outlier problem and how granularity is…
Read article →HQQ: Half-Quadratic Quantization Explained
HQQ (half-quadratic quantization): data-free low-bit weight quantization solved in seconds, an outlier-aware objective, and how it compares to GPTQ an…
Read article →INT4 Kernel Architecture, in depth: W4A16 versus W4A4, packing, register dequantisation and the decode-to-prefill crossover
How 4-bit matrix-multiply kernels for LLM inference are built: why weight-only INT4 speeds up decode, W4A16 versus W4A8 versus W4A4 and which GPUs hav…
Read article →KV Cache Quantization Architecture in Depth
A 2500-word walkthrough of KV cache quantization: FP16 baseline, INT8/INT4/FP8 formats, per-head scales, write/read paths, kernels, accuracy, runtime.
Read article →LLM Quantization Architecture in Depth
LLM quantization as a serving architecture: why decode is bandwidth bound, INT4, FP8, MXFP4 and NVFP4 formats, GPTQ, AWQ and SmoothQuant, fused kernel…
Read article →LLM.int8(): Outlier Features and Mixed-Precision Matmul
LLM.int8() explained: why outlier features break 8-bit quantization past ~6.7B parameters, vector-wise int8, and the fp16 decomposition for outlier co…
Read article →Marlin INT4 Kernel: 4-Bit Matmul on GPU Tensor Cores Explained
How the Marlin INT4xFP16 kernel uses GPU tensor cores: packed 4-bit weights with group scales, async loads, register dequantization, MMA and split-K.
Read article →Mixed-precision inference architecture
Deep-dive on mixed-precision LLM serving: calibration and per-layer sensitivity profiling, planner-assigned FP16/FP8/INT8/INT4 manifests, kernel regis…
Read article →NF4 Quantization and QLoRA: 4-bit NormalFloat Fine-Tuning
How NF4 and QLoRA work: NormalFloat4 levels at normal quantiles, block-wise scales, double quantization, LoRA adapters on a frozen 4-bit base, paged o…
Read article →Quantization-aware training
Deep-dive on QAT: fake quantization in the forward pass, straight-through estimator gradients, comparison to post-training quantization, learned quant…
Read article →QuaRot Explained: Hadamard Rotation for Outlier-Free INT4
The rotation idea behind QuaRot and SpinQuant: a Hadamard rotation spreads activation outliers, folds into the weights, so 4-bit activations quantize …
Read article →SmoothQuant, in depth: where the scales fold, how calibration feeds them, and a W8A8 pipeline you can ship
The architecture of SmoothQuant W8A8 quantization: why activation outliers block INT8, where the smoothing scales fold inside a decoder block, the cal…
Read article →SpinQuant architecture
Deep-dive on SpinQuant: why outlier channels wreck naive INT4 quantization, how orthogonal rotations spread outliers without changing the full-precisi…
Read article →Symmetric vs asymmetric quantization architecture
Deep-dive on symmetric versus asymmetric quantization: how a float range maps onto integer levels through a scale and a zero-point, why symmetric quan…
Read article →Weight clustering quantization
Deep-dive on weight clustering (codebook / weight-sharing) quantization, which compresses a model by running k-means over each layer's weights to…
Read article →Quantization Deep Dive: How 4-bit and 1.5-bit Models Retain 99% of Their Original Accuracy
The immense power of Large Language Models (LLMs) comes with a significant burden: their colossal size.
Read article →Quantization Evaluation Methodology, in depth: perplexity, KL divergence, flips and paired statistics for judging a quantized model
How to evaluate a quantized LLM against its own baseline: pinning the setup, comparable perplexity, KL divergence and top-1 agreement from logits, tas…
Read article →Quantization for Attention, in depth: low-precision QK and PV matmuls, smoothing K, rotations and when it pays
How to quantize the attention computation itself rather than weights or the KV cache: where attention FLOPs dominate, why K has channel outliers and w…
Read article →Quantization for Embeddings, in depth: float16, int8 and binary vectors, rescoring, calibration and measuring the recall you lose
How to store embedding vectors in fewer bytes without losing search quality: why embeddings quantize differently from model weights, memory arithmetic…
Read article →SmoothQuant Intuition, in depth: why one channel ruins a token, the seesaw identity, and what alpha really trades
A first-principles mental model of SmoothQuant: how outlier channels inflate the INT8 step for every token, why per-channel activation scales cannot l…
Read article →