All 56 articles, sorted alphabetically
AWQ vs GPTQ Quantization
Two ways to get 4-bit weights — and when each one wins.
Read article →bitsandbytes vs AutoAWQ vs AutoGPTQ
Library comparison for INT4 quant.
Read article →FP8 Inference Explained
H100-era format with INT8-like speed and FP-like dynamic range.
Read article →GGUF Format Explained
The on-disk format powering llama.cpp and local inference.
Read article →GPU Kernels for INT4 Inference
Why hardware support is moving the bar.
Read article →INT8 vs INT4 Quantization
What you actually lose and what you gain.
Read article →INT8 Calibration for LLMs
SmoothQuant and the activation outlier story.
Read article →Mixed-Precision Inference
FP16 BF16 FP8 INT4 in one model.
Read article →AWQ
How AWQ identifies and protects important weights based on activation magnitudes, achieving strong INT4 quality.
Read article →BF16 (Brain Float)
How BF16 preserves FP32's dynamic range with less precision, and why it's standard for LLM training.
Read article →FP16 (Half Precision)
How FP16 provides 2x smaller / faster with modest quality loss for many workloads.
Read article →GGUF
The GGUF format used by llama.cpp for quantized model distribution, supporting many precision variants.
Read article →GPTQ
How GPTQ quantizes trained models layer-by-layer with calibration data, minimizing quality loss.
Read article →INT4 Quantization
How INT4 quantization delivers 8x memory savings, and the techniques (GPTQ, AWQ, GGUF) that make it viable.
Read article →INT8 Quantization
How INT8 quantization delivers 4x memory savings with minor quality loss for LLM inference.
Read article →LLM.int8()
How LLM.int8() enables INT8 quantization by isolating outlier features and computing them in FP16.
Read article →Quantization Overview
What quantization is, why it matters for LLM inference, and the precision options available.
Read article →SmoothQuant
How SmoothQuant makes activation quantization viable by shifting difficulty from activations to weights.
Read article →QLoRA Fine-Tuning Explained
4-bit base + LoRA adapter = fine-tune big models cheaply.
Read article →Activation quantization architecture
Deep-dive on activation quantization: per-token/per-tensor scales, outlier isolation, static vs dynamic, kernel support, W4A8/W8A8.
Read article →Adaptive rounding architecture
Deep-dive on adaptive rounding (AdaRound): why round-to-nearest is optimal for the wrong objective, layer-wise reconstruction on calibration activatio…
Read article →AWQ architecture
Deep-dive on AWQ: calibration, salient channel detection, scale search, int4 group quantization, kernels, evaluation, comparison.
Read article →AWQ vs GPTQ Quantization Architectures Compared
A 2500-word walkthrough of AWQ vs GPTQ quantization architectures: calibration, algorithm, output format, kernel support, and trade-offs.
Read article →Bit packing architecture
Deep-dive on bit packing for int4 weights: pack layout, dense byte storage, unpack kernel, fused dequant + matmul, hardware fit.
Read article →BitNet ternary quantization architecture
Deep-dive on BitNet b1.58: ternary {-1,0,+1} weights and int8 activations, the BitLinear layer, absmean/absmax quantizers, straight-through training, …
Read article →Quantization calibration -- choosing the right ranges
Deep-dive on quantization calibration: the float-to-integer range problem, representative calibration data, range selection (min/max vs percentile), o…
Read article →Double quantization architecture
Deep-dive on double (nested) quantization: block-wise quantizing weights to low bits with per-block floating-point scales, then quantizing those scale…
Read article →Dynamic vs static quantization architecture
Deep-dive comparing dynamic and static post-training quantization: offline weight quantization shared by both, static calibration that bakes activatio…
Read article →FP8 -- 8-bit floating point for training and inference
Deep-dive on FP8: the E4M3 and E5M2 formats, floating point vs integer (dynamic range vs uniform steps), scaling, native tensor-core support (Hopper/B…
Read article →GGUF quantization architecture
Deep-dive on GGUF and llama.cpp quantization: file layout and metadata, block quantization with per-block scales, Q4_K_M super-blocks, IQ2/IQ3 with im…
Read article →GPTQ architecture
Deep-dive on GPTQ post-training quantization: calibration, Hessian, Cholesky, column quant, error redistribution, group size.
Read article →Quantization granularity -- per-tensor, per-channel, per-group scales
Deep-dive on quantization granularity: per-tensor vs per-channel vs per-group scales, scale and zero-point, the outlier problem and how granularity is…
Read article →Half-Quadratic Quantization (HQQ) architecture
Deep-dive on HQQ: quantization as optimization with a sparse outlier term, half-quadratic splitting into closed-form proximal and zero-point steps, da…
Read article →INT4 kernel architecture
Deep-dive on INT4 GEMM kernels: Marlin, Machete, dequant path, shape support, scale layout, async pipeline, benchmarks.
Read article →KV Cache Quantization Architecture in Depth
A 2500-word walkthrough of KV cache quantization: FP16 baseline, INT8/INT4/FP8 formats, per-head scales, write/read paths, kernels, accuracy, runtime.
Read article →KV cache quantization architecture
Deep-dive on KV cache quantization: calibration, per-channel + per-token scaling, RoPE handling, fused kernels, and eval harness.
Read article →LLM Quantization Architecture in Depth
A 2500-word walkthrough of LLM quantization: FP16 base, calibration, GPTQ/AWQ/SmoothQuant/FP8 methods, weight/activation/KV formats, fused kernels, ev…
Read article →LLM.int8() architecture
Deep-dive on LLM.int8(): why systematic outlier features break plain 8-bit quantization past ~6.7B parameters, vectorwise int8 quantization, column-sp…
Read article →Marlin INT4 kernel architecture
Deep-dive on the Marlin INT4xFP16 matmul kernel: packed weight layout and group scales, async double-buffered loads, register-level dequantization, te…
Read article →Mixed-precision inference architecture — sensitivity profiling, per-layer precision plans, and fused-kernel serving
Deep-dive on mixed-precision LLM serving: calibration and per-layer sensitivity profiling, planner-assigned FP16/FP8/INT8/INT4 manifests, kernel regis…
Read article →NF4 and QLoRA -- 4-bit quantization for fine-tuning
Deep-dive on NF4 and QLoRA: the fine-tuning memory problem, NormalFloat4 (levels at normal-distribution quantiles matching weight distributions), bloc…
Read article →Quantization-aware training
Deep-dive on QAT: fake quantization in the forward pass, straight-through estimator gradients, comparison to post-training quantization, learned quant…
Read article →Rotation-based quantization architecture
Deep-dive on rotation-based quantization: why activation outliers break four-bit inference, how an orthogonal Hadamard rotation spreads outlier energy…
Read article →SmoothQuant architecture
Deep-dive on SmoothQuant: activation outlier detection, smoothing factor, weight scaling, INT8 quantize both, calibration, kernels.
Read article →SpinQuant architecture
Deep-dive on SpinQuant: why outlier channels wreck naive INT4 quantization, how orthogonal rotations spread outliers without changing the full-precisi…
Read article →Symmetric vs asymmetric quantization architecture
Deep-dive on symmetric versus asymmetric quantization: how a float range maps onto integer levels through a scale and a zero-point, why symmetric quan…
Read article →Weight clustering quantization
Deep-dive on weight clustering (codebook / weight-sharing) quantization, which compresses a model by running k-means over each layer&a…
Read article →Quantization-Aware Training
When QAT beats post-training.
Read article →Quantization Deep Dive: How 4-bit and 1.5-bit Models Retain 99% of Their Original Accuracy
Read article →Quantization Evaluation Methodology
Beyond perplexity — task-specific eval.
Read article →Quantization for Attention
KV cache attention scores special cases.
Read article →Quantization for Embeddings
Scalar binary product quantization.
Read article →SmoothQuant Intuition
Why migrating outliers works.
Read article →Speculative Decoding
Use a small model to draft, big model to verify.
Read article →