Quantization

Quantization

INT8/INT4, GGUF, AWQ, GPTQ, SmoothQuant, FP8 KV cache.

56Articles
56Topics covered
Articles in this category

All 56 articles, sorted alphabetically

Advertisement
ARTICLE · 01

AWQ vs GPTQ Quantization

Two ways to get 4-bit weights — and when each one wins.

Read article
ARTICLE · 02

bitsandbytes vs AutoAWQ vs AutoGPTQ

Library comparison for INT4 quant.

Read article
ARTICLE · 03

FP8 Inference Explained

H100-era format with INT8-like speed and FP-like dynamic range.

Read article
ARTICLE · 05

GGUF Format Explained

The on-disk format powering llama.cpp and local inference.

Read article
ARTICLE · 06

GPU Kernels for INT4 Inference

Why hardware support is moving the bar.

Read article
ARTICLE · 08

INT8 vs INT4 Quantization

What you actually lose and what you gain.

Read article
ARTICLE · 09

INT8 Calibration for LLMs

SmoothQuant and the activation outlier story.

Read article
ARTICLE · 10

Mixed-Precision Inference

FP16 BF16 FP8 INT4 in one model.

Read article
ARTICLE · 11

AWQ

How AWQ identifies and protects important weights based on activation magnitudes, achieving strong INT4 quality.

Read article
ARTICLE · 12

BF16 (Brain Float)

How BF16 preserves FP32's dynamic range with less precision, and why it's standard for LLM training.

Read article
ARTICLE · 13

FP16 (Half Precision)

How FP16 provides 2x smaller / faster with modest quality loss for many workloads.

Read article
ARTICLE · 14

GGUF

The GGUF format used by llama.cpp for quantized model distribution, supporting many precision variants.

Read article
ARTICLE · 15

GPTQ

How GPTQ quantizes trained models layer-by-layer with calibration data, minimizing quality loss.

Read article
ARTICLE · 16

INT4 Quantization

How INT4 quantization delivers 8x memory savings, and the techniques (GPTQ, AWQ, GGUF) that make it viable.

Read article
ARTICLE · 17

INT8 Quantization

How INT8 quantization delivers 4x memory savings with minor quality loss for LLM inference.

Read article
ARTICLE · 18

LLM.int8()

How LLM.int8() enables INT8 quantization by isolating outlier features and computing them in FP16.

Read article
ARTICLE · 19

Quantization Overview

What quantization is, why it matters for LLM inference, and the precision options available.

Read article
ARTICLE · 20

SmoothQuant

How SmoothQuant makes activation quantization viable by shifting difficulty from activations to weights.

Read article
ARTICLE · 21

QLoRA Fine-Tuning Explained

4-bit base + LoRA adapter = fine-tune big models cheaply.

Read article
ARTICLE · 22

Activation quantization architecture

Deep-dive on activation quantization: per-token/per-tensor scales, outlier isolation, static vs dynamic, kernel support, W4A8/W8A8.

Read article
ARTICLE · 23

Adaptive rounding architecture

Deep-dive on adaptive rounding (AdaRound): why round-to-nearest is optimal for the wrong objective, layer-wise reconstruction on calibration activatio…

Read article
ARTICLE · 24

AWQ architecture

Deep-dive on AWQ: calibration, salient channel detection, scale search, int4 group quantization, kernels, evaluation, comparison.

Read article
ARTICLE · 25

AWQ vs GPTQ Quantization Architectures Compared

A 2500-word walkthrough of AWQ vs GPTQ quantization architectures: calibration, algorithm, output format, kernel support, and trade-offs.

Read article
ARTICLE · 26

Bit packing architecture

Deep-dive on bit packing for int4 weights: pack layout, dense byte storage, unpack kernel, fused dequant + matmul, hardware fit.

Read article
ARTICLE · 27

BitNet ternary quantization architecture

Deep-dive on BitNet b1.58: ternary {-1,0,+1} weights and int8 activations, the BitLinear layer, absmean/absmax quantizers, straight-through training, …

Read article
ARTICLE · 28

Quantization calibration -- choosing the right ranges

Deep-dive on quantization calibration: the float-to-integer range problem, representative calibration data, range selection (min/max vs percentile), o…

Read article
ARTICLE · 29

Double quantization architecture

Deep-dive on double (nested) quantization: block-wise quantizing weights to low bits with per-block floating-point scales, then quantizing those scale…

Read article
ARTICLE · 30

Dynamic vs static quantization architecture

Deep-dive comparing dynamic and static post-training quantization: offline weight quantization shared by both, static calibration that bakes activatio…

Read article
ARTICLE · 31

FP8 -- 8-bit floating point for training and inference

Deep-dive on FP8: the E4M3 and E5M2 formats, floating point vs integer (dynamic range vs uniform steps), scaling, native tensor-core support (Hopper/B…

Read article
ARTICLE · 32

GGUF quantization architecture

Deep-dive on GGUF and llama.cpp quantization: file layout and metadata, block quantization with per-block scales, Q4_K_M super-blocks, IQ2/IQ3 with im…

Read article
ARTICLE · 33

GPTQ architecture

Deep-dive on GPTQ post-training quantization: calibration, Hessian, Cholesky, column quant, error redistribution, group size.

Read article
ARTICLE · 34

Quantization granularity -- per-tensor, per-channel, per-group scales

Deep-dive on quantization granularity: per-tensor vs per-channel vs per-group scales, scale and zero-point, the outlier problem and how granularity is…

Read article
ARTICLE · 35

Half-Quadratic Quantization (HQQ) architecture

Deep-dive on HQQ: quantization as optimization with a sparse outlier term, half-quadratic splitting into closed-form proximal and zero-point steps, da…

Read article
ARTICLE · 36

INT4 kernel architecture

Deep-dive on INT4 GEMM kernels: Marlin, Machete, dequant path, shape support, scale layout, async pipeline, benchmarks.

Read article
ARTICLE · 37

KV Cache Quantization Architecture in Depth

A 2500-word walkthrough of KV cache quantization: FP16 baseline, INT8/INT4/FP8 formats, per-head scales, write/read paths, kernels, accuracy, runtime.

Read article
ARTICLE · 38

KV cache quantization architecture

Deep-dive on KV cache quantization: calibration, per-channel + per-token scaling, RoPE handling, fused kernels, and eval harness.

Read article
ARTICLE · 39

LLM Quantization Architecture in Depth

A 2500-word walkthrough of LLM quantization: FP16 base, calibration, GPTQ/AWQ/SmoothQuant/FP8 methods, weight/activation/KV formats, fused kernels, ev…

Read article
ARTICLE · 40

LLM.int8() architecture

Deep-dive on LLM.int8(): why systematic outlier features break plain 8-bit quantization past ~6.7B parameters, vectorwise int8 quantization, column-sp…

Read article
ARTICLE · 41

Marlin INT4 kernel architecture

Deep-dive on the Marlin INT4xFP16 matmul kernel: packed weight layout and group scales, async double-buffered loads, register-level dequantization, te…

Read article
ARTICLE · 42

Mixed-precision inference architecture — sensitivity profiling, per-layer precision plans, and fused-kernel serving

Deep-dive on mixed-precision LLM serving: calibration and per-layer sensitivity profiling, planner-assigned FP16/FP8/INT8/INT4 manifests, kernel regis…

Read article
ARTICLE · 43

NF4 and QLoRA -- 4-bit quantization for fine-tuning

Deep-dive on NF4 and QLoRA: the fine-tuning memory problem, NormalFloat4 (levels at normal-distribution quantiles matching weight distributions), bloc…

Read article
ARTICLE · 44

Quantization-aware training

Deep-dive on QAT: fake quantization in the forward pass, straight-through estimator gradients, comparison to post-training quantization, learned quant…

Read article
ARTICLE · 45

Rotation-based quantization architecture

Deep-dive on rotation-based quantization: why activation outliers break four-bit inference, how an orthogonal Hadamard rotation spreads outlier energy…

Read article
ARTICLE · 46

SmoothQuant architecture

Deep-dive on SmoothQuant: activation outlier detection, smoothing factor, weight scaling, INT8 quantize both, calibration, kernels.

Read article
ARTICLE · 47

SpinQuant architecture

Deep-dive on SpinQuant: why outlier channels wreck naive INT4 quantization, how orthogonal rotations spread outliers without changing the full-precisi…

Read article
ARTICLE · 48

Symmetric vs asymmetric quantization architecture

Deep-dive on symmetric versus asymmetric quantization: how a float range maps onto integer levels through a scale and a zero-point, why symmetric quan…

Read article
ARTICLE · 49

Weight clustering quantization

Deep-dive on weight clustering (codebook / weight-sharing) quantization, which compresses a model by running k-means over each layer&a…

Read article
ARTICLE · 50

Quantization-Aware Training

When QAT beats post-training.

Read article
ARTICLE · 51

Quantization Deep Dive: How 4-bit and 1.5-bit Models Retain 99% of Their Original Accuracy

Read article
ARTICLE · 52

Quantization Evaluation Methodology

Beyond perplexity — task-specific eval.

Read article
ARTICLE · 53

Quantization for Attention

KV cache attention scores special cases.

Read article
ARTICLE · 54

Quantization for Embeddings

Scalar binary product quantization.

Read article
ARTICLE · 55

SmoothQuant Intuition

Why migrating outliers works.

Read article
ARTICLE · 56

Speculative Decoding

Use a small model to draft, big model to verify.

Read article