Why it matters

Batched GEMM shapes LLM speed. Understanding shapes optimization.

Advertisement

The architecture

N (M x K) x (K x N) matmuls.

Grouped GEMM: varying shapes.

Strided batched: uniform shapes.

Batched GEMM typesUniform batchedsame shapesGrouped GEMMvarying shapesKernel launchamortizedcuBLAS + CUTLASS + CUB provide batched primitives; MoE needs grouped
Batched GEMM math.
Advertisement

How it works end to end

cuBLAS strided batched.

CUTLASS grouped GEMM for MoE.

Kernel launch amortization.