Why it matters

Intensity shapes perf regime. Understanding shapes optimization.

Advertisement

The architecture

M x K by K x N: 2 M N K FLOPs + M K + K N + M N bytes.

Intensity high for M+N+K large.

Matmul arithmetic intensity2 M N K FLOPscomputeM K + K N + M N bytesmemoryFLOP/byte ratio= intensityH100 balance point ~300 FLOP/byte bf16; large matmul above, small below
Matmul intensity.
Advertisement

How it works end to end

H100 balance ~300 FLOP/byte bf16.

Large matmul compute-bound.

Small matmul BW-bound.

Batching helps small.