Why architecture matters here
Quantization granularity matters because it's a primary determinant of quantized model accuracy, especially at low bit-widths where outliers dominate. The whole point of quantization is to reduce size/cost while preserving accuracy, and granularity is a key lever: too coarse (per-tensor) and outliers crush precision (a large scale to accommodate outliers means the common values are poorly represented -- accuracy loss), finer (per-channel, per-group) and outliers are isolated (their effect contained, common values well-represented -- accuracy preserved). At higher bit-widths (int8), per-tensor may suffice (enough precision to tolerate the outlier-driven scale); at lower bit-widths (int4), finer granularity (per-channel, per-group) is often essential (the precision is too scarce to waste on accommodating outliers with a coarse scale). So granularity choice directly affects whether aggressive quantization (int4) maintains acceptable accuracy -- finer granularity is often what makes low-bit quantization viable. For anyone quantizing models (especially aggressively), understanding granularity -- and choosing finer granularity to handle outliers -- is essential to preserving accuracy, and it's why modern low-bit quantization schemes (GPTQ, AWQ, etc.) use fine granularity (per-channel or per-group).
The outlier insight is the architectural crux, and it's why granularity matters so much for LLMs. LLM weights and especially activations have outliers -- a small number of values far larger in magnitude than the rest (in activations, certain dimensions consistently have large values). Quantization maps the value range to the low-bit integer range via the scale, and the scale must accommodate the full range (including outliers). With a coarse scale (per-tensor -- one scale for the whole tensor), the outliers force a large scale (to represent the large outlier values), which means the many normal values (small relative to the outliers) map to only a few of the available quantization levels (their precision crushed -- most of the low-bit range is 'used up' by the outlier magnitude). This is the outlier problem: outliers, via a coarse shared scale, destroy the precision of the common values. Finer granularity solves it: with a per-channel scale, an outlier in one channel forces a large scale only for that channel (the other channels have their own scales, unaffected -- their normal values well-represented); with per-group, even finer (the outlier's effect contained to its group). So finer granularity isolates outliers, containing their range-inflating effect to a small scope (a channel or group), preserving the precision of the common values elsewhere. This -- finer granularity isolating outliers to preserve common-value precision -- is why granularity is central to quantization accuracy (especially given LLMs' outliers), and understanding the outlier problem explains why finer granularity helps.
And the accuracy-versus-overhead-versus-kernel-support tradeoff is what makes granularity a real design decision, not just 'finer is better'. Finer granularity (per-group over per-channel over per-tensor) generally improves accuracy (better outlier handling), but it has costs. Metadata overhead: more scales to store (per-tensor: one scale; per-channel: one per channel; per-group: one per group -- many more) -- so finer granularity uses more memory for the quantization metadata (though usually small relative to the weights, it's non-zero, and very fine granularity -- tiny groups -- adds up). Kernel support: the hardware/kernels must handle the granularity efficiently -- a kernel doing the dequantization/computation must apply the right scale per element, and finer granularity (many scales) is more complex for the kernel (per-tensor is simple -- one scale; per-group needs the kernel to apply the right group's scale per weight). Not all hardware/kernels support all granularities efficiently, so the granularity choice is constrained by kernel support (using a granularity the target hardware handles efficiently). So the granularity choice trades accuracy (finer better) against metadata overhead (finer more) and kernel support (finer more complex, constrained by hardware) -- a real decision balancing accuracy needs, overhead, and hardware compatibility. Understanding that granularity trades accuracy against overhead and kernel support -- and choosing the granularity balancing these for the accuracy target and target hardware -- is what makes granularity a design decision rather than a simple 'finest possible' choice.
The architecture: every piece explained
Top row: the levels and parameters. Per-tensor: one scale for the entire tensor (simplest, least metadata -- one scale -- but a single scale for the whole tensor's range, so outliers force a large scale crushing common-value precision). Per-channel: a scale per channel (row or column of the weight matrix -- each channel's range handled separately, so an outlier in one channel doesn't hurt others -- much better outlier handling, one scale per channel of metadata). Per-group: a scale per group/block of weights (e.g., every 128 weights within a channel -- even finer, handling local range variation within channels -- best outlier handling, one scale per group of metadata -- the finest common granularity, used by low-bit schemes). Scale and zero-point: the quantization parameters -- scale (the multiplier mapping integers to floats) and zero-point (an offset, for asymmetric quantization -- allowing the quantized range to not be centered at zero) -- per the granularity (per-tensor: one scale/zero-point; per-channel: per channel; per-group: per group).
Middle row: outliers and the tradeoffs. Outliers: the reason granularity matters -- outlier values (far larger than the rest) force a large scale with coarse granularity, crushing common-value precision; finer granularity isolates outliers (containing their range effect to a channel/group). Accuracy vs overhead: finer granularity is more accurate (better outlier handling) but has more metadata (more scales) -- the core tradeoff. Symmetric vs asymmetric: symmetric quantization (no zero-point -- the range centered at zero, simpler, for ranges roughly symmetric like weights) vs asymmetric (with a zero-point -- for ranges not centered at zero, like activations after ReLU which are non-negative) -- the quantization scheme, interacting with granularity. Weights vs activations: different strategies -- weights (static, quantized once -- can afford fine granularity, per-channel/group) vs activations (dynamic, vary per input -- often per-tensor or per-token, and outliers are a bigger problem, motivating techniques like SmoothQuant to shift outliers to weights) -- the granularity and scheme differing for weights vs activations.
Bottom rows: kernel and size. Kernel support: the hardware/kernels must handle the granularity efficiently -- applying the right scale per element (per-tensor: simple; per-group: the kernel applies the right group's scale per weight -- more complex) -- so the granularity choice is constrained by what the target hardware/kernels support efficiently. Group size choice: for per-group, the group size (128, 64, 32 weights per group) -- smaller groups (finer -- better accuracy, more metadata, more kernel complexity) vs larger (coarser -- less metadata, simpler, slightly less accurate) -- a tradeoff within per-group (128 is common, balancing accuracy and overhead). The ops strip: granularity choice (choosing the granularity -- per-tensor/channel/group and group size -- balancing accuracy needs, overhead, and kernel support for the bit-width and hardware), kernel compatibility (ensuring the target hardware/kernels efficiently support the chosen granularity -- a constraint), and accuracy (measuring the quantized accuracy -- finer granularity for better accuracy where the coarser loses too much).
End-to-end flow
Trace granularity's effect on accuracy. A model is quantized to int4 (aggressive -- scarce precision). With per-tensor granularity (one scale for the whole tensor): the tensor has outliers (some weights far larger), so the single scale is large (to represent the outliers), and the many normal weights map to only a few int4 levels (their precision crushed) -- the quantized model's accuracy is poor (the common weights, most of the model, poorly represented). Switching to per-channel (a scale per channel): each channel's outliers force a large scale only for that channel (the other channels unaffected -- their normal weights well-represented) -- accuracy improves significantly (outliers isolated per channel). Going further to per-group (a scale per 128-weight group): even the local range variation within channels is handled (each group's scale fitting its weights) -- accuracy improves more (the best outlier handling). The granularity determined the accuracy: per-tensor was too coarse (outliers crushing precision), per-channel much better (outliers isolated), per-group best (finest handling) -- demonstrating that finer granularity handles outliers better, preserving accuracy at the aggressive int4 bit-width.
The kernel-support and group-size vignettes show the practical constraints. A kernel case: the team wants per-group int4 (best accuracy), but the target hardware's kernels must support per-group efficiently (applying the right group's scale per weight during the dequantization/matmul) -- they verify the target inference stack (e.g., a kernel library supporting per-group int4 -- as GPTQ/AWQ kernels do) handles it efficiently, so the per-group granularity is both accurate and performant on the target hardware. Had the target lacked efficient per-group support, they'd have used a coarser granularity the hardware handles (a kernel-support constraint on the granularity choice). A group-size case: for per-group, they choose the group size -- 128 (the common choice, balancing accuracy and overhead), versus 64 (finer -- slightly better accuracy, more metadata, more kernel work) or 256 (coarser -- less accurate, less overhead) -- selecting 128 as the accuracy-overhead balance for their needs (finer where accuracy demands, coarser where overhead matters).
The weights-vs-activations and measurement vignettes complete it. A weights-vs-activations case: the team uses fine granularity (per-channel or per-group) for weights (static, quantized once -- can afford the fine granularity), but for activations (dynamic, outliers a bigger problem), they use a technique like SmoothQuant (shifting the activation outliers into the weights, so activations are easier to quantize) plus appropriate activation granularity -- the different strategies for weights (fine granularity) vs activations (outlier-shifting plus suitable granularity). A measurement case: they measure the quantized model's accuracy (on real tasks) at each granularity -- confirming per-group achieves acceptable int4 accuracy (where per-tensor didn't) -- validating the granularity choice by measured accuracy (not assuming). The consolidated discipline the team documents: understand granularity (per-tensor/channel/group -- how finely scales are applied) as a primary accuracy determinant (finer isolates outliers, preserving common-value precision -- essential at low bit-widths), choose finer granularity where accuracy demands (per-channel/group for int4), balance against metadata overhead and kernel support (constrained by hardware -- use a granularity the target handles efficiently), choose group size for the accuracy-overhead balance (128 common), use different strategies for weights (fine granularity) and activations (outlier-shifting plus suitable granularity), and measure accuracy to validate -- because quantization granularity determines how well quantization handles the outliers that dominate low-bit accuracy, and choosing finer granularity (balanced against overhead and kernel support) is often what makes aggressive quantization maintain acceptable accuracy.