Why it matters

Grouped GEMM enables efficient MoE. Understanding shapes kernel design.

Advertisement

The architecture

Multiple (M, N, K) matmuls in one call.

Per-group parameters.

Grouped GEMMGroup 1: (M1, N, K)expert 1 tokensGroup 2: (M2, N, K)expert 2 tokensBatched launchone kernelcuBLAS + TransformerEngine + CUTLASS + Triton implementations
Grouped GEMM.
Advertisement

How it works end to end

cuBLAS: cublasGemmGroupedBatched.

CUTLASS: templated.

TransformerEngine grouped GEMM for FP8.