Why architecture matters here

H100 utilization fails on missed TMA opportunities, wrong tensor core precision, or poor occupancy. Architecture matters because kernel + shape + precision decide MFU.

Advertisement

The architecture: every piece explained

The top strip is compute. SM count 132. Warp schedulers 4 per SM. CUDA cores 128 per SM. Tensor cores 4 per SM 4th gen.

The middle row is memory + async. Register file 256KB / SM. Shared memory + L1 256KB / SM. TMA async copy bulk transfers. Distributed shared across cluster launches.

The lower rows are practice. FP8 + INT8 MMA. Occupancy — warps in flight. Ops — tuning + MFU + scheduling.

H100 SM — warp scheduler + tensor cores + shared memory + async copythe compute unit of HopperSM count132 on H100 SXMWarp schedulers4 per SMCUDA cores128 per SMTensor cores4 per SM 4th genRegister file256KB / SMShared memory + L1256KB / SMTMA async copybulk transfersDistributed sharedcluster launchFP8 + INT8 MMAtensor core precisionsOccupancywarps in flightOps — kernel tuning + MFU + schedulingregisterssmemasyncclusterprecisionsoccupancyoccupancyoperateoperate
H100 SM internal structure.
Advertisement

End-to-end flow

End-to-end: attention kernel launches with thread block cluster. TMA loads K + V blocks async to shared memory. Tensor cores execute FP8 GEMM. Distributed shared memory shares between cluster blocks. Occupancy 60%. MFU 55%. Profile via Nsight confirms.