Why architecture matters here

Kernel choice fails on shape mismatch, wrong scale layout, or missing async support. Architecture matters because kernel + hardware + shape are joint.

Advertisement

The architecture: every piece explained

The top strip is the tools. Quantized weights int4 packed. Marlin Ampere+. Machete Hopper W4A16. Dequant path in-kernel to bf16.

The middle row is details. Shape support M/N/K. Scale layout group vs channel. Async pipeline TMA. Benchmarks vs fp16.

The lower rows are ops. Kernel selection at load. Failure modes fallback. Ops — matrix per (arch, dtype, shape).

INT4 kernels — Marlin + Machete + dequant path + shapes + benchmarksget the throughput int4 promisesQuantized weightsint4 packedMarlin kernelAmpere+ optimizedMachete kernelHopper W4A16Dequant pathin-kernel to bf16Shape supportM/N/K constraintsScale layoutgroup vs channelAsync pipelineTMA on HopperBenchmarksvs fp16 baselineKernel selectionat model loadFailure modesshape / dtype fallbackOps — matrix per (arch, dtype, shape) + testsmatchlayoutasyncmeasureselectfallbackfallbackoperateoperate
INT4 kernel landscape and dequant path.
Advertisement

End-to-end flow

End-to-end: 70B model quantized int4 with GPTQ group=128. Loaded on H100. Machete kernel selected for W4A16. TMA loads weights async. Throughput 2.4x fp16 baseline. Benchmarks confirm.