Every GPU optimization argument eventually becomes an argument about where the data is. A thread can reach four qualitatively different places for a value: its own registers, the shared memory and L1 sitting inside its SM, the L2 cache shared by the whole chip, and HBM out past the memory controllers. Those four levels differ from each other by orders of magnitude in capacity, and by comparable orders of magnitude in bandwidth and latency — running in opposite directions. The level with room for your model has the least bandwidth; the level with the most bandwidth has room for almost nothing. This article treats the hierarchy as one picture rather than four separate features, and turns it into a single practical question: where should this kernel’s working set live?
The whole hierarchy in one picture
Draw the four levels as a stack and two properties move monotonically. Going outward from the thread — registers, then shared memory and L1, then L2, then HBM — capacity grows and bandwidth shrinks. Latency grows with it. That is not a coincidence of a particular product; it is what physical distance and wire count buy you. Registers sit inside the datapath, so they are absurdly fast and correspondingly tiny. HBM is DRAM on the package, reachable only through memory controllers, so it holds the model but every access is a long trip.
Keep that figure as the mental model. Two of the four levels are yours to control (registers, via the compiler; shared memory, via your code) and two are hardware-managed. Optimization is mostly the work of getting hot data into the levels you control and keeping it there.
Registers — the widest and closest storage there is
The register file is the fastest storage on the chip and, counterintuitively, one of the largest pools of SRAM on an SM — because it is replicated per thread. Each thread gets a private slice, allocated by the compiler; there is no ‘load from register’ instruction because operands come from there by default. Bandwidth here is effectively the issue rate of the ALUs, which is why nothing else in the hierarchy is in the same conversation.
The catch is that the register file is a fixed resource split across every resident thread. Ask for more registers per thread and fewer thread-blocks fit on the SM, which reduces the number of warps available to hide latency. That tradeoff — register budget against residency, and the spilling that happens when you overrun it — belongs to the GPU occupancy architecture article. Here the point is that level one is per-thread, private, and rationed.
Shared memory and L1 — the same silicon, two personalities
One step out, an SM has a block of SRAM that serves two roles: a hardware-managed L1 data cache, and a software-managed scratchpad visible to every thread in a block. On NVIDIA parts these are the same physical storage, partitioned between the two uses, which is why claiming a large scratchpad shrinks the cache.
This is the first level where you decide what lives there. A cache guesses at reuse from access history; a scratchpad lets you assert it. If you know a tile of a matrix will be read by every thread in the block, you stage it there once and read it many times, instead of hoping the cache keeps it. That explicit control is why shared memory is the pivot of nearly every hand-tuned GPU kernel. Its internals — banks, conflicts, and the padding tricks that avoid them — belong to the GPU shared memory architecture article; treat it here as the first level you can place data in deliberately.
L2 — the GPU-wide backstop you cannot address
Beyond the SMs sits a single L2 shared by the entire GPU, and it is the level people forget because there is no API for it. Every global-memory access that misses in L1 goes through it, from every SM, and so does traffic you did not initiate: copy engines touch it too. It is the level where blocks that landed on different SMs share data cheaply.
Its practical effect is to raise the bandwidth you observe without changing HBM at all. If a grid’s blocks read overlapping regions, the second and later reads can be served on-chip. This is why launch configurations that make concurrently-running blocks touch nearby addresses beat ones that scatter across the whole tensor. You cannot pin data in L2 the way you pin it in shared memory, but you can arrange your access pattern so L2 has a chance.
HBM — capacity at the end of a wire
HBM is where the model, the activations, and the KV cache actually live. It is stacked DRAM on the same package as the die, connected by an extremely wide interface — that width is the entire reason its bandwidth is measured in terabytes per second rather than the hundreds of gigabytes a conventional DIMM interface delivers. Compared to on-chip SRAM, though, it is still slow and still far.
Two consequences follow. First, capacity planning is an HBM question: what fits determines batch size, sequence length, and whether a model needs sharding across devices. Second, HBM bandwidth is the resource most LLM inference workloads run out of first — decode reads the full weight set per step and does very little arithmetic with it. The stacking and the generational differences are internals covered by the dedicated HBM article; what matters for placement is that this is the level of last resort, and every trip out here should earn its keep.
The shape of the curve — relative, not absolute
Exact capacities and bandwidths change every generation and differ per SKU, so the useful version of this table is relative. The magnitudes below are illustrative, not a spec sheet — check the programming guide for the part you actually run on.
| Level | Scope | Managed by | Relative capacity | Relative bandwidth |
|---|---|---|---|---|
| Registers | One thread | Compiler | Smallest | Highest — datapath speed |
| Shared / L1 | One thread-block / one SM | You / hardware | Kilobytes per SM | An order below registers |
| L2 | Whole GPU | Hardware | Tens of megabytes class | An order below shared |
| HBM | Whole GPU | You, via allocation | Tens of gigabytes class | Lowest — the bottleneck |
Read the last two columns together and the design problem is obvious: capacity and bandwidth are anti-correlated at every step. No level is both big and fast, so a kernel touching a lot of data must move it rather than find it already close.
The reuse argument — a fetch has to pay for itself
Here is the argument that makes the hierarchy actionable. Divide the GPU’s operations per second by its HBM bytes per second and you get its machine balance: how many arithmetic operations it can perform in the time it takes to fetch one byte. On current datacenter parts that ratio is large — compute has grown far faster than memory for years.
So a value pulled from HBM must be used many times before the fetch breaks even. Use it once and the ALUs sit idle waiting for the next byte; the kernel is memory-bound and only moving fewer bytes will speed it up. The ratio of operations to bytes moved is arithmetic intensity, and comparing it against machine balance is the roofline check. The hierarchy is how you raise that ratio: every level you promote a value to makes the next read of it cheaper.
Tiling — how a working set climbs the hierarchy
Tiling is the standard mechanism for turning one HBM fetch into many uses. Take a matrix multiply. The naive form has each output element read a full row and a full column, so every input is pulled from HBM repeatedly and arithmetic intensity is dismal. The tiled form cuts the output into blocks; each thread-block stages a tile of A and a tile of B into shared memory, and every thread then reads those tiles many times from on-chip storage.
The bytes crossing the HBM boundary drop by roughly the tile dimension, and the fetch that does happen now serves a whole block of work. Stage it one level further — accumulators and operand fragments in registers — and reuse climbs again. Tiling is fractal in exactly this way: the same trick applies between HBM and shared, and between shared and registers. FlashAttention is this reasoning applied to attention, and has its own article.
Deciding where the working set should live
The placement question has a workable procedure. First, name the working set: the data touched repeatedly within one unit of work. Second, size it. If it fits in registers per thread, keep it there and stop. If it fits in the per-SM scratchpad for a whole block, stage it there and read it many times. If it fits in L2 across the concurrently-running blocks, index those blocks so they touch neighbouring addresses. If it only fits in HBM, restructure until an earlier answer becomes true.
Restructuring means changing the unit of work, not micro-optimizing loads: smaller tiles, blocked loops, a different traversal order, fusing consecutive operations so an intermediate never round-trips to HBM. That last one is kernel fusion, covered separately. The discipline is asking the sizing question before tuning anything, because a working set parked at the wrong level cannot be rescued by better instructions.
When the hierarchy cannot save you
Some workloads have no reuse to exploit. An elementwise add over two large tensors touches every byte exactly once; no amount of tiling creates reuse that the algorithm does not contain. For these, the hierarchy is irrelevant and the only levers are moving fewer bytes — fusing neighbouring elementwise ops, using a narrower dtype — and making sure each byte moved arrives efficiently, which is coalescing, covered by its own article.
LLM decoding is the important case of this. Each step streams the weights and the KV cache from HBM and performs comparatively little arithmetic per byte, so it is memory-bound almost by construction. That is why serving strategies attack it by increasing reuse across requests — batching more sequences so one weight fetch serves many tokens. The same reuse argument, applied one level up from the kernel to the scheduler.