Evidence label: deterministic synthetic model. The lab is a JavaScript model of weight-only quantization. It generates synthetic weights for seven layers, including rare large weights, and synthetic activations with a few outlier channels, then actually rounds each weight row in groups to a 3, 4 or 8-bit grid, optionally after activation-aware scaling, and measures each layer's output error on the synthetic activations. The perplexity figure is an illustrative mapping from those errors, not a measurement of any model.
Play the lab
How to play
Choose the bit width and group size, then move the activation-aware scaling slider (alpha) and toggle the zero point, 8-bit down-projection and calibration data. Every change re-quantizes all seven layers with the same random seed. Watch the per-layer bars: the goal is not only a low average but no layer far above the rest. The tiles show estimated perplexity, worst and mean layer error, bits per weight including scale overhead, and the size of a 7B model built with the same settings.
Missions unlock in order and reward the cheapest configuration that meets each target. The events change the model or its traffic: a traffic shift that activates outlier channels the calibration data never saw, and a fine-tuned checkpoint with more large weights.
What the model assumes
Weights are near-Gaussian with a small fraction of large values; the down-projection has more of them and a wider input, as is common in decoder models. Activations are random per token with fixed per-channel magnitudes, and a few channels are 10 to 45 times larger than the rest. Quantization is round-to-nearest within each group, with a symmetric or asymmetric grid and FP16 scales. Activation-aware scaling uses the calibration set's mean channel magnitude raised to alpha. Perplexity is modelled as FP16 perplexity multiplied by an exponential of the parameter-weighted squared output errors.
What it cannot tell you
There is no GPTQ-style error compensation, no clipping search, no per-layer alpha search and no interaction between layers: errors in real models propagate and sometimes cancel through the residual stream. Real outlier structure differs between architectures and checkpoints. Use the lab to understand why some layers break and which knob addresses which cause, then measure your own model with per-layer errors and task evaluations.
Read the full explanation
- Why INT4 Breaks Some Layers: Outliers, Group Size and Activation-Aware Scaling (companion article)
- LoRA Rank, Alpha and Target Modules: Choosing Settings That Actually Work
- Compute-Optimal Model Sizing: Applying Chinchilla Scaling Laws to a Real Budget
All AI in Production labs · Lab catalog
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- vLLM documentation: quantization
- Hugging Face Transformers documentation: quantization overview
Views are my own and do not represent any employer, past or present.