The H100 is the part that most of the last few years of large-model work actually ran on, and it is quoted more often than it is understood. A spec sheet gives you a memory capacity, a bandwidth number and a headline TFLOPS figure, and every one of those three is conditional on which H100 you bought, which numeric format you use, and whether the vendor counted structured sparsity. This article takes the H100 at the level of the whole part rather than the level of a kernel: what is on the GH100 die and how much of it is enabled, how memory and cache are provisioned, what NVLink buys that PCIe cannot, and which capabilities — the Transformer Engine, MIG, confidential computing, DPX — are properties of the product rather than of any one instruction. The interior of a streaming multiprocessor, where warp scheduling, tensor cores and the Tensor Memory Accelerator live, has its own article; everything here sits outside that boundary.
What the name H100 actually covers
H100 names a family, not a single configuration. Every member is built from the same GH100 die, but they differ in how much of that die is enabled, what kind of memory is stacked next to it, how much power the board is allowed to draw, and whether the chip can talk to its peers over NVLink at full rate. Those differences are not cosmetic. Two parts both truthfully labelled H100 can differ by more than a third in memory bandwidth and by a factor of two in board power, and a benchmark result quoted without the form factor is close to meaningless.
The mainstream configurations are the SXM5 module, which is what goes into an 8-GPU baseboard; the PCIe card, which drops into a conventional server slot; and the NVL variant, a paired PCIe part aimed at inference on large models. Alongside them sit the GH200 Grace Hopper superchip, which packages an H100 with an Arm-based Grace CPU over a coherent chip-to-chip link, and the H200, which keeps H100 compute and replaces the memory system.
The practical consequence is that almost every question of the form how fast is an H100 needs a qualifier. Throughout this article, unqualified numbers describe the SXM5 module, because that is the part nearly all large-scale training and most serious serving actually runs on, and the PCIe differences are called out where they matter.
GH100 on the die - GPCs, TPCs, and the SMs that ship
GH100 is a large chip: roughly 80 billion transistors on a TSMC 4N process tuned for NVIDIA. Its top-level organisation is a hierarchy. The die is divided into graphics processing clusters (GPCs) — eight of them — each holding a set of texture processing clusters (TPCs), and each TPC holding a pair of streaming multiprocessors. Fully populated, that arrangement yields 144 SMs.
No shipping H100 has 144. The SXM5 module enables 132; the PCIe card enables 114. This is ordinary yield harvesting: on a die this large, some SMs will be defective, and disabling whole TPCs lets a partially defective die be sold rather than scrapped. It also gives NVIDIA a product lever, since the gap between 132 and 114 is a real performance difference sold at a lower price in a lower-power package. Practically, it means SM count is a per-SKU fact to look up, not a per-architecture constant, and any capacity model that scales linearly in SM count must use the number for the part actually racked.
The GPC boundary matters less to a kernel author than the SM boundary — work is distributed to SMs, not to GPCs — with one significant exception. Hopper's thread-block clusters guarantee co-residency on a group of physically adjacent SMs, and that adjacency is a GPC-level property. The cluster mechanism itself belongs to the SM article; the die-level point is simply that the physical grouping the feature depends on is visible here.
Five stacks of HBM3, and why the number is 80 GB
Memory on an H100 is not a chip on a board. It is a set of HBM stacks sitting on the same package as the die, connected through a silicon interposer. GH100 provides six stack sites; the 80 GB H100 populates and enables five of them, at 16 GB each, giving a 5120-bit-wide path to memory. That is where 80 GB comes from, and it is why the capacity is not a round power of two — it is five stacks, not four or eight.
Width is the whole design argument. HBM does not run at exotic clock rates; it wins by being absurdly wide and physically close, and the H100 SXM5 turns that into roughly 3.3 TB/s of peak bandwidth. What makes that figure the most important number on the sheet is the arithmetic of transformer inference: decoding one token from a model whose weights do not fit in cache requires streaming those weights out of HBM, so single-stream decode latency is set by bandwidth and essentially not at all by TFLOPS. The compute number governs prefill and training; the bandwidth number governs generation.
Capacity is the other half, and it binds differently. Eighty gigabytes has to hold weights, optimiser state or KV cache, activations and workspace at once — which is why a model that would fit on paper often does not fit in practice, and why parallelism strategy is usually forced by capacity before it is chosen for speed. How DRAM stacks work internally, and why achieved bandwidth always trails peak, is covered in this category's dedicated HBM article.
L2 and the shared middle of the chip
Between 132 SMs and five stacks of memory sits the L2 cache, and on Hopper it is a product-level lever rather than an implementation detail. Each SM has its own L1 and scratchpad, but L2 is the only cache every SM shares, so it is where a working set touched by many blocks either lands or does not. H100's L2 is substantially larger than the 40 MB A100 carried, and that growth is deliberate: as arithmetic gets cheaper faster than memory gets faster, the cheapest way to raise effective bandwidth is to serve more requests without leaving the die.
The structural caveat is that the cache is physically partitioned across the chip rather than being one flat pool. An access finds its data in the nearer partition or the farther one, and the difference is real even though nothing in CUDA exposes it directly. That is one reason locality helps at scales larger than a thread block: keeping cooperating blocks working on nearby addresses keeps traffic on the near side of the on-die network.
Hopper also gives programs partial control over what L2 keeps. Access policy windows let a kernel mark a region as persisting, biasing the cache toward holding a small hot structure — an embedding slice, a lookup table, a set of shared parameters — instead of letting a large streaming tensor evict it. It is a hint, not a pin, and it only pays when the hot region is genuinely small relative to the cache. The general theory of the hierarchy, from registers down to HBM, is treated separately; what belongs here is that L2 capacity is one of the things that changed between generations, and it changed for a reason.
SXM5 versus PCIe - two products, one die
The form factor is the single most consequential choice in H100 procurement, and it is routinely elided. An SXM5 module is not a card: it is a mezzanine module that bolts into a vendor baseboard, which handles power delivery and cooling and carries the NVLink traces between sockets. It enables more SMs, pairs the die with HBM3, and is allowed to draw up to about 700 W. A PCIe card enables fewer SMs, pairs the die with HBM2e at meaningfully lower bandwidth, and lives inside a 350 W envelope that a standard server slot and its airflow can actually sustain.
Power is the hinge. Roughly doubling the board budget is what allows higher sustained clocks across more SMs and a faster memory type, which is why the SXM part is not a modest step up but a different performance class. It is also why the SXM part is not something you retrofit: 700 W per socket, eight sockets to a baseboard, is a rack-power and cooling problem before it is a compute decision.
The interconnect difference is starker still. SXM modules get the full NVLink complement into a switched fabric; PCIe cards get, at most, a bridge linking a pair of cards, with everything else routed over the host bus. Any workload that shards a model across more than two GPUs and exchanges tensors every layer feels that gap directly.
The NVL variant sits between them, pairing two bridged PCIe-class parts with more HBM3 per card than the standard PCIe SKU, specifically so a large model fits in the pair. The heuristic is durable: tightly coupled multi-GPU training wants SXM; single-GPU or loosely coupled inference is well served by PCIe, and often better served per dollar.
NVLink 4 and NVSwitch - the scale-up domain as a product feature
NVLink is what makes a group of H100s behave like one large accelerator rather than eight separate ones. Each SXM5 module carries 18 fourth-generation links, aggregating to the widely quoted 900 GB/s counting both directions. Set that against a PCIe Gen5 x16 host connection and the ratio is roughly seven to one — and the ratio, not either absolute figure, is what decides which parallelism strategies are viable.
The critical detail is that those links do not go GPU-to-GPU in a fixed pattern on a standard baseboard. They terminate in NVSwitch chips, so the eight GPUs form a non-blocking all-to-all group: any GPU can reach any other at full link bandwidth, and no pair is topologically privileged. That uniformity is what lets tensor parallelism ignore placement inside a node, and it is why the eight-GPU baseboard became the unit of deployment rather than the single card.
Beyond the node, external NVLink switching can extend a single NVLink domain past one chassis, while the far more common arrangement is NVLink inside the node and InfiniBand or RoCE between nodes. Either way the shape of the problem is the same: there is a bandwidth cliff at the edge of the NVLink domain, and good placement puts the chattiest parallelism — tensor parallel — inside it, with pipeline and data parallelism spanning the slower boundary. NVSwitch topology, collective bandwidth and how domains are sized have their own articles in this category.
The Transformer Engine and FP8 as a shipped capability
Hopper's most commercially significant addition is eight-bit floating point on the matrix pipe, and it arrives as a product capability rather than merely a datatype. The hardware supports two FP8 encodings with different trades between exponent range and mantissa precision, and doubles matrix throughput relative to 16-bit formats while halving what weights and activations cost in bytes — memory and bandwidth as well as flops.
The problem eight bits creates is dynamic range. A single FP8 format cannot span the range that appears across a training run, so values must be scaled into the representable window per tensor, and the right scale drifts as training proceeds. The Transformer Engine is the library-plus-hardware answer: it tracks the magnitudes a tensor has recently taken, chooses scaling factors from that history, keeps accumulation and master weights in wider precision, and decides per operation whether FP8 is safe. The user-visible effect is that FP8 becomes a configuration flag instead of a numerical research project.
Two caveats belong at the product level. First, the speedup is bounded by what fraction of a model's time is spent in large matrix multiplications; a workload dominated by memory movement or small operations sees little. Second, FP8 is not free accuracy — it is a controlled loss with well-tested recipes, which is a different claim. The bit layouts, the scaling algorithms and the amax history that drives them are worked through in the transformer-math article on FP8 training.
MIG - carving one part into isolated instances
An H100 is frequently too large for the job in front of it. A small model served at modest request rates will leave most of 132 SMs idle while still occupying the whole board, and time-slicing several tenants onto one GPU trades that waste for unpredictable latency, since each tenant's tail depends on the others' bursts.
Multi-Instance GPU is the hardware answer: the part is partitioned into as many as seven instances, each with its own slice of SMs, its own slice of memory and its own path to that memory. The memory path is the part that matters. Because the partitioning reaches into the cache and memory controllers rather than just scheduling kernels in turn, one instance's bandwidth-hungry job cannot starve another's — which is the property that makes an SLA on a shared GPU defensible at all.
The costs are equally structural. Instances are created from a fixed menu of profile geometries with placement constraints, so the partition you want may not be expressible; reconfiguration is disruptive rather than instantaneous, which makes MIG a provisioning decision rather than an autoscaling knob; and a partitioned GPU cannot run one large job, so capacity stranded in the wrong shape stays stranded. On H100 the mechanism is second-generation, with each instance able to participate in the part's confidential-computing mode. Profile geometry, the comparison against MPS, and which workloads actually fit a slice are covered in this category's MIG article.
Confidential computing on a GPU
H100 was the first NVIDIA datacenter part able to join a trusted execution environment, which addresses a specific commercial blocker: an organisation that will not let its weights or its customers' data appear in cleartext on infrastructure it does not own. Before Hopper, CPU-side enclaves stopped at the PCIe bus, and the moment work moved to the accelerator the guarantee lapsed.
The mechanism has three parts. A protected region of GPU memory is fenced off by hardware so that the host, other devices and other tenants cannot read it, with the CPU's access to that region blocked rather than merely discouraged. Data crossing the PCIe boundary in either direction is encrypted, so the bus itself is not a disclosure channel. And the GPU can produce a signed attestation of its identity and firmware state, which a relying party checks before releasing keys — so the trust decision is made against evidence rather than against a provider's assurance. Describing the feature as simply encrypted GPU memory, as shorthand summaries often do, misses both the isolation model and the attestation step that makes the whole thing useful.
The performance shape follows from where the cryptography sits. It is on the transfer path, not on the compute path, so a job that ships a large input, computes for a long time and returns a small output pays very little, while a chatty workload moving tensors host-to-device constantly pays much more. Confidential mode is enabled at the driver level and changes what the platform can do, so it is a deployment-time decision, not a per-process one.
DPX and the workloads that are not transformers
It is easy to read H100 as a transformer accelerator with some legacy attached, and that reading gets procurement wrong. Hopper also added DPX instructions, which accelerate the inner loop shared by a large family of dynamic-programming algorithms: the fused pattern of combining candidate values with a minimum or maximum, adding a cost term, and clamping the result. That pattern is the recurrence in sequence alignment used across genomics, in all-pairs shortest-path graph algorithms, and in route and scheduling optimisation. Before Hopper it compiled into a short chain of general instructions per cell; afterwards it is fewer instructions with the saturation behaviour handled in hardware.
The point is not the specific speedup, which depends heavily on how well the recurrence is tiled. It is that these workloads are latency-bound on a dependency chain rather than throughput-bound on matrix math, so they were historically poor GPU candidates. Shortening the chain changes that calculus.
The same qualification applies to double precision. H100 retains substantial FP64 capability, including a matrix path for it — a genuine differentiator against consumer and inference-oriented parts, where FP64 is deliberately crippled. A cluster bought for language models but also expected to run classical simulation is buying that capability whether or not anyone priced it, and a cheaper part chosen on AI-format throughput alone may be an order of magnitude worse at the simulation work.
What changed versus A100, and what Blackwell changed again
Placing H100 between its neighbours makes the design intent legible. Against A100, the changes cluster into four groups. Scale: more SMs on a newer process at a higher power budget. Memory: HBM3 in place of HBM2e, which is most of a doubling in bandwidth, plus a larger L2. Interconnect: NVLink moved from 600 GB/s to 900 GB/s aggregate and the host link from PCIe Gen4 to Gen5. And capability: FP8 with the Transformer Engine, thread-block clusters with distributed shared memory, the Tensor Memory Accelerator, DPX, and confidential computing — none of which A100 has at all. The first three make old code faster; the fourth requires new code, which is why the realised gap between the two parts varies so widely by software stack.
Against Blackwell, the pattern repeats one level up. Blackwell's headline structural move is packaging two large dies behind a very wide die-to-die link and presenting them to CUDA as a single device, which sidesteps the reticle limit that capped GH100's size. On top of that sit substantially more HBM3e capacity and bandwidth per device, a doubled NVLink generation, and a second-generation Transformer Engine that pushes below eight bits into FP4 for inference. The consequences for planning are that the memory-capacity ceiling that forced sharding on H100 moves outward, and that per-GPU power moves into territory where liquid cooling stops being optional.
Meanwhile H200 occupies a narrower slot: identical Hopper compute, a memory system rebuilt around HBM3e for materially more capacity and bandwidth in the same socket. On compute-bound work it is barely distinguishable from H100; on bandwidth-bound decoding it is a large step. That is a clean natural experiment demonstrating which half of the spec sheet a given workload actually lives on.
Reading an H100 spec sheet without being fooled
Four habits prevent most of the misquotation. First, always attach the form factor. A bandwidth or throughput figure without SXM or PCIe attached is unreliable by a wide margin, and the gap is wider than most people assume because the memory type differs, not just the clocks.
Second, check for the sparsity asterisk. NVIDIA's largest published tensor numbers assume the 2:4 structured-sparsity feature, which doubles the quoted rate but requires the model's weights to have been pruned into that pattern and validated. Most production models are dense, so the applicable figure for most people is half the biggest number on the page. Comparing one vendor's sparse peak against another's dense peak is the most common way GPU comparisons go wrong.
Third, state the format. Throughput roughly doubles at each step down the ladder from TF32 to FP16 and BF16 to FP8, while FP64 sits an order of magnitude or more below the AI formats. A TFLOPS number with no datatype attached carries almost no information.
Fourth, discount peak to achieved. Real large-model training sustains a fraction of peak — a good result is roughly half, and workloads that are bandwidth-bound or communication-bound land far lower. Capacity plans built on peak numbers under-provision by a factor that only shows up after the hardware arrives. The honest planning input is a measured throughput figure for your own model on the exact SKU, and everything above is context for why the measurement came out where it did.