Why architecture matters here
The architectural reason clustering earns its place is that it decouples the number of distinct values a layer uses from the precision those values are stored in. A codebook can hold sixteen very precise fp16 centroids while the weights themselves cost only four bits each to reference. This is a fundamentally different knob from uniform quantization, which ties value resolution directly to bit width. Because the two are decoupled, clustering can preserve the exact magnitudes of the few values that matter while spending almost nothing per weight, which is exactly the regime where extreme compression without total collapse becomes possible.
The data distribution of neural weights is what makes this pay off. Trained weights are overwhelmingly small and clustered near zero, with a sparse set of large-magnitude outliers that carry disproportionate importance. Uniform quantization, forced to cover the outlier range with evenly spaced levels, ends up with coarse resolution in the dense center where most of the action is. A learned codebook allocates its centroids by density — many near zero, a few far out — so it captures both the crowded center and the important tail. Matching the representation to the distribution is the whole game, and clustering does it by construction rather than by heuristic scaling.
Fine-tuning is what turns clustering from lossy rounding into recoverable compression, and it is why the technique belongs in a training-adjacent pipeline rather than as a pure post-processing filter. Once weights are tied to shared centroids, you can continue training with the constraint that all weights in a cluster move together — gradients for every weight in a cluster are accumulated and applied to the shared centroid. The network re-learns to do its job using only K values per layer, and this retraining routinely recovers most or all of the accuracy that naive clustering would have lost. The centroids become learned parameters, not just statistical summaries.
Granularity is the next architectural lever. A single codebook for an entire layer is the most compact but the least expressive; per-channel or per-block codebooks give each region its own set of centroids, which fits local weight distributions far better at the cost of storing more small codebooks. The choice mirrors the per-tensor versus per-channel decision in uniform quantization: finer granularity buys accuracy and costs metadata. Where you land depends on how heterogeneous the layer's weights are and how tight the storage budget is, and it is worth tuning per model rather than assuming a global default.
Finally, the inference cost model is what determines whether clustering is the right tool at all. The index-then-gather step adds a memory indirection that pure integer quantization avoids, and hardware without native support for the gather pattern may not turn the storage savings into speed. Clustering is therefore strongest when memory footprint and bandwidth — not arithmetic throughput — are the binding constraint: on-device deployment, models that must fit in limited RAM, or bandwidth-bound serving where fewer bytes moved per weight directly buys latency. Framing the decision around the actual bottleneck avoids adopting clustering for a workload where a simpler integer scheme would have been faster.
The architecture: every piece explained
The clustering step is the front of the pipeline: for each layer (or channel, or block, depending on granularity) run k-means over the weights to find K centroids. Initialization matters — placing initial centroids by density or linearly across the range affects both convergence and which values survive — and the large-magnitude outliers deserve special care so they are not averaged away into the dense center. The output of this step is a set of K representative values and an assignment of every weight to its nearest centroid.
The codebook stores those K centroid values, typically in full or half precision because there are so few of them that their storage cost is negligible. This is the crucial asymmetry: the codebook is small and precise, so it can afford to represent the important magnitudes faithfully, while the per-weight cost lives entirely in the index. A layer might have millions of weights but only 16 or 256 codebook entries, so the effective bits-per-weight is dominated by the index width, not the centroid precision.
The index map is the bulk of the compressed layer: one index per weight, each log2(K) bits wide, pointing into the codebook. This is where the compression actually lives — replacing a 16-bit float with a 4-bit index is a 4x reduction before any other trick. The index map is packed tightly (bit-packed indices), and its width is the single biggest lever on the compression ratio: halving K shaves a bit off every weight in the model.
The fine-tuning loop is the accuracy-recovery component. After clustering, training resumes with weights constrained to their shared centroids: forward and backward passes run normally, but gradients are summed within each cluster and applied to the shared value, so all weights sharing a centroid stay tied. Over a few epochs the centroids adjust and the network adapts to its reduced vocabulary, recovering accuracy that pure post-training clustering would leave on the table. This loop is optional for mild compression and essential for aggressive compression.
The inference-time gather is how a packed layer is served. For each weight the runtime reads its index, looks up the centroid in the codebook, and feeds the resulting value into an otherwise standard matmul — the model computes as if the weights were full precision, because after the gather they effectively are. Whether this is fast depends on hardware support for the gather and on whether the codebook fits in fast memory. The diagram shows the full path from original weights through clustering, codebook, index map, and optional fine-tuning to the packed, servable layer.
End-to-end flow
The pipeline starts with a trained full-precision (or already lightly quantized) model. For each target layer, the tooling extracts the weight tensor and decides the clustering granularity — one codebook for the whole tensor, or one per output channel or block. This decision is made up front because it determines how many k-means runs happen and how much codebook metadata the final model carries.
k-means then runs on each unit of weights: centroids are initialized, weights are assigned to their nearest centroid, centroids are recomputed as the mean of their members, and the loop repeats to convergence. The result is K centroids and a full assignment. At this point the layer is already conceptually compressed — every weight is representable by an index — but the accuracy may have dropped, especially if K is small or outliers were mishandled during initialization.
If the pipeline includes fine-tuning, training resumes with the clustering constraint active. Each forward pass uses the current centroids as the effective weights; each backward pass computes per-weight gradients, but instead of updating weights individually, the gradients within each cluster are accumulated and applied to the shared centroid. Over several epochs the centroids drift to better positions and the network recovers, often reaching accuracy indistinguishable from the original at a fraction of the storage. This step is where most of the quality of a well-clustered model is decided.
Once accuracy is acceptable, the layer is packed: the codebook is serialized in full precision, the assignments are bit-packed into an index map of log2(K)-bit entries, and any per-channel codebooks are stored with their scope. The packed model is dramatically smaller on disk and in memory than the original — this is the artifact you ship to a device or load into a memory-constrained server. Metadata records the granularity and K per layer so the runtime knows how to unpack.
At inference, the runtime loads codebook and index map into memory. For each matmul, it gathers centroid values through the indices — either materializing the dequantized weight tile on the fly or fusing the gather into the kernel — and performs the standard multiply-accumulate. The codebook is small enough to sit in fast memory, so the gather is a cheap lookup, and the model produces the same outputs it would with full-precision weights up to the clustering error. The net effect is far less memory moved and stored per weight, which on bandwidth-bound hardware translates directly into lower latency and a smaller footprint.