Why architecture matters here

Bit packing failures come from layout mismatch. Architecture matters because layout + kernel + hardware combine.

Advertisement

The architecture: every piece explained

The top strip is packing. int4 weights. Pack layout. Dense byte array. Metadata.

The middle row is unpack. Unpack kernel. Fused dequant + matmul. Memory bandwidth. Hardware fit.

The lower rows are practice. Interleaving patterns. Checkpoint format. Ops — kernel matrix.

Bit packing — dense storage + unpack kernel + memory bandwidth + hardware fitfit int4 into bytes efficientlyint4 weightshalf-bytePack layoutaligned + interleavedDense byte arrayno paddingMetadatascales + zero pointsUnpack kernelshift + maskFused dequant + matmulin-registerMemory bandwidthcut in halfHardware fitPTX / SIMD intrinsicsInterleaving patternsK/N or blockCheckpoint formatportable + versionedOps — kernel matrix + eval + rollbackunpackfusemeasuretargetpatternversionversionoperateoperate
Bit packing layout + unpack in fused GEMM kernel.
Advertisement

End-to-end flow

End-to-end: 4-bit weights packed 2-per-byte. Interleaved along K for coalesced load. Kernel loads bytes; SIMD shifts unpack 4-bit pairs; dequant via scale + zero point; matmul in tensor cores. Memory bandwidth halved; throughput 2x fp16.