CPU KERNELS · SIMD LANES · INT8 DOT PRODUCT
SIMD Register — AVX-512 + AMX
Run one u8 x s8 dot product through scalar, AVX-512, AVX2, VNNI and AMX instruction shapes and watch lanes fill, saturate and accumulate.
Your experiment
The weights are signed bytes of magnitude 64 to 127 whose sign flips every second weight, and the activations are unsigned bytes up to a chosen maximum. Start on the AVX2 path with activations up to 200, then lower the maximum, switch to VNNI, float32 and scalar, and give AMX a full 16 by 16 tile.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
A model of instruction semantics and counts, not a CPU benchmark: no cycles, latencies, memory traffic, tile configuration or data layout are modelled, and loads, conversions and horizontal reductions are not counted. Lane arithmetic follows the documented behaviour of each instruction; the data come from a fixed seed.
One dot product, five instruction shapes
The lab computes a dot product of 256 unsigned 8-bit activations with 256 signed 8-bit weights, the inner loop of an int8 matrix multiply, on five instruction shapes; Instruction path picks the one that runs, and Vector length sets how many elements the dot product has. A scalar loop does one multiply-add per instruction. AVX-512 vfmadd231ps converts to float32 and does 16 per instruction. AVX2 vpmaddubsw multiplies 32 byte pairs and adds adjacent products into 16-bit lanes, followed by vpmaddwd and vpaddd. AVX-512 VNNI vpdpbusd multiplies 64 byte pairs and adds each group of four into 16 int32 lanes. AMX tdpbusd multiplies a 16 x 64 byte activation tile by a weight tile holding 64 x 16 bytes into a 16 x 16 int32 tile.
Saturation in 16-bit lanes
On the opening AVX2 path with Largest activation value at 200, 11 of 128 adjacent-pair sums exceed 32,767 and vpmaddubsw clamps them, so the result is 58,597 instead of the exact 60,649: SATURATED: WRONG RESULT. Adjacent weights share a sign in this fixture, so pairs add rather than cancel. With activations up to 128 the largest possible pair is 128 x 127 x 2 = 32,512, which fits, and the result is exact; in this data 150 is still exact and 160 clamps 2 pairs. VNNI accumulates in 32 bits and gives 60,649 at any setting here.
Counting instructions
For one output of length 256 the compute instructions are: scalar 256, float32 FMA 16, AVX2 24 (8 groups of three), VNNI 4 and AMX 4, that is 1, 16, 10.67, 64 and 64 useful multiply-adds per instruction. Every count grows in proportion to Vector length. Loads, conversions and the final horizontal reductions are not counted. The float32 path is exact here: float32 represents integers exactly up to 16,777,216, and with 1,024 elements and activations up to 255 no lane sum comes close.
AMX needs a full tile
With 1 activation row and 1 weight column, each tdpbusd performs 16,384 multiply-adds of which 64 are useful: 0.39% tile utilization. Set Activation rows and Weight columns to 16 and the same 4 instructions compute 256 outputs at 100% utilization, while VNNI needs 1,024 instructions for the same work. Tiles pay off for batched matrix multiplication, not for a single matrix-vector product.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the setup, the lane arithmetic of the first instruction, saturation or rounding checks, the running total after instructions 1, 2 and the last, the instruction count and the final comparison across sixty inspection seconds; the table shows the first eight products and the running exact sum. Instruction counts are not cycle counts: latency, throughput, memory bandwidth, tile configuration and data layout decide real speed and are not modelled.