Why architecture matters here

GPTQ fails on wrong calibration or too aggressive bits. Architecture matters because Hessian + column order + groups compose.

Advertisement

The architecture: every piece explained

The top strip is math. Calibration set. Layer input. Hessian H. Cholesky.

The middle row is algorithm. Column-wise quant. Error redistribute. 4-bit weights. Accuracy check.

The lower rows are ops. Group size. Metrics. Ops — calibration + kernel + serving.

GPTQ — Hessian + layer-wise + reconstructionpost-training quantization for LLMsCalibration setsmall samplesLayer inputactivationsHessian HX^T X approxCholeskyH^-1Column-wise quantsequentialError redistributeto remaining4-bit weightsgroup-wiseAccuracy checkvs FP16Group size128 typicalMetricsperplexity deltaOps — calibration + kernel + servingquantspreadpackverifysizewatchwatchoperateoperate
GPTQ quantizes columns sequentially, redistributing error.
Advertisement

End-to-end flow

End-to-end: load FP16 model. Pass 128 calibration samples. Per layer, compute H = X^T X. Cholesky-invert. Quantize column j, compute residual, redistribute to columns > j using H^-1 row. Repeat. Group size 128 lets scales adapt.