Why it matters
Every large model's training speed is bounded by tensor core throughput. GPT-scale models train in weeks instead of years because tensor cores exist. Understanding them lets you write code that actually uses them.
The architecture
Tensor cores compute D = A × B + C where A, B, C, D are small matrices (4x4, 8x8, or 16x16 depending on generation). Each operation completes in one clock cycle — hundreds of times faster than the same work on CUDA cores.
Precision modes: FP16 (half precision), BF16 (brain float, better for training), FP8 (Hopper+), INT8 (inference), TF32 (default Ampere+). Lower precision means more throughput.
How it works end to end
Engagement conditions: matrix operations with compatible sizes (multiples of 8 or 16), correct data types, aligned memory. Frameworks handle this automatically; hand-written kernels must use WMMA or MMA intrinsics.
Mixed precision training: compute forward and backward in FP16/BF16, keep master weights in FP32. Standard technique for large model training.
Sparsity: Ampere+ tensor cores support 2:4 structured sparsity for 2x throughput on sparse matrices.