Why architecture matters here
Tensor core failures come from picking the wrong dtype path (falling back to CUDA cores), wrong tile size, or missing sparsity opportunities. Architecture matters because kernel choice + dtype align the workload to hardware.
The architecture: every piece explained
The top strip is the primitive. Kernel launcher sets up GEMM. WMMA / MMA is the warp-level matrix multiply-accumulate. Tile size depends on architecture. Accumulator dtype typically fp32 for stability.
The middle row is the details. Input dtype bf16/fp16/fp8. Sparsity 2:4 structured for 2x on Ampere+. Multi-buffering hides memory latency. Kernel selection uses cuBLAS/cuDNN/CUTLASS.
The lower rows are ops. Nsight profiling for MFU per kernel. Failure modes — dtype fallback. Ops covers hardware matrix + library version.
End-to-end flow
End-to-end: training loop launches GEMM. Kernel picks CUTLASS with bf16 input + fp32 accum + tile 128x128. WMMA executes; multi-buffering hides HBM read latency. Nsight shows 60% MFU. Sparsity 2:4 applied to weights; 40% speedup on inference. Kernel matrix versioned per driver.