Cooling design in an AI hall is not really a temperature question — every tier on the menu can hold a die at a safe junction temperature. It is a density question: how many kilowatts you must pull out of one rack footprint, and which medium can still carry that much away. Each rung of the ladder — air, rear-door heat exchanger, cold plate, immersion — exists because the rung below it ran out of capacity. This piece walks the ladder in order, then the loop behind it: the CDU, primary and secondary water, approach temperature, fluid chemistry, leak discipline, heat reuse, and why PUE reads better for liquid than the physics strictly earns.
Density, not temperature, is the design variable
A single accelerator has never been hard to cool in isolation. What changed is how many are bolted into one rack and how much power each draws. A traditional enterprise hall was engineered around racks in the single-digit kilowatts; a rack of dense accelerator nodes can ask for an order of magnitude more in the same floor tile. The plant does not care that the load is GPUs; it cares that heat flux per square metre of white space rose by a factor you cannot fan your way out of.
So: you do not pick a cooling technology, you pick a density, and the density picks the technology. The electrical side of that same density — feeders, busway, redundancy, and the step-load behaviour of synchronised training jobs — is its own subject; here the watts are a given, and the question is how they leave the building.
Why accelerator TDP growth broke air cooling
Air is a poor coolant and the arithmetic is unforgiving. Heat removed equals mass flow times specific heat times temperature rise. Air carries roughly 1.2 kJ per cubic metre per kelvin; water carries around 4,180 — some 3,500 times more per unit volume. Pulling 30 kW out of a rack with a 12 K rise therefore needs on the order of 2.5 m³/s of air, several thousand CFM, through one cabinet. Double the load and you must double the flow or the delta-T, and both have walls: inlet temperature is bounded below by chiller economics and above by component reliability and throttling.
Worse, fan power follows the affinity laws and rises roughly with the cube of flow. Chasing the last increment of air-cooled density burns a fast-growing share of the rack's own power budget on moving air, and that fan energy is itself heat you must then remove. Air hit a knee where each extra kilowatt got disproportionately expensive.
The ladder and the thresholds that force each step
The bands below are planning rules of thumb, not standards; real thresholds shift with containment quality and plant supply temperature. The ordering, though, is stable even when the numbers move.
| Tier | Typical band | What limits it |
|---|---|---|
| Raised floor, no containment | single-digit kW | Recirculation, hot spots |
| Aisle containment, CRAH | ~10-20 kW | Airflow volume, fan power |
| Rear-door heat exchanger | ~30-50 kW | Water temperature, coil area |
| Cold plate (hybrid) | ~50-150 kW | Residual air load, flow per node |
| Immersion | highest, no fans | Serviceability, weight, fluid |
Read it as a sequence of capitulations: each rung moves the working fluid closer to the silicon — from the room, to the back of the rack, to the lid of the package, to the package itself.
Rear-door heat exchangers — the retrofit bridge
A rear-door heat exchanger is a water-cooled coil hung on the back of an otherwise conventional rack. Server fans still push air front-to-back; the door catches that exhaust and dumps its heat into water before it re-enters the room. A passive door relies entirely on the servers' own fans and adds pressure drop they must overcome; an active door adds fans of its own, at the cost of power and another failure domain.
The appeal is that nothing inside the server changes — standard chassis, maintenance, spares. Done well the rack becomes roughly thermally neutral to the room, which is what lets an existing hall absorb densities its CRAH units could never have handled. The limits are the coil's face area and the water temperature: colder water buys capacity but costs chiller energy, and the door can never remove heat the server fans failed to carry out.
Direct-to-chip cold plates — the mainstream answer
Direct liquid cooling stops using air as an intermediary for the hot components. A cold plate — a metal block with fine internal channels — clamps onto the accelerator and CPU packages, and coolant is manifolded to every node. Because water is thousands of times better per unit volume than air, flow rates are small enough for tubing rather than ducting, and the fan power that was eating the rack's budget largely disappears.
The practical point is that cold-plate systems are hybrid, not liquid-only. Plates capture the large, hot, well-located heat sources; voltage regulators, memory, NICs, drives, and power supplies keep dumping their share into air. A hall that plans for that residual air load — rear-door coils, or a reduced but genuine air plant — behaves; one that assumed liquid took everything ends up with an undersized air path and unhappy peripherals. The selection question is simply what fraction of rack heat the plate captures.
Immersion — single-phase and two-phase
Immersion drops whole boards into a bath of dielectric fluid. In single-phase immersion the fluid stays liquid and circulates through an external heat exchanger; convection does the work. In two-phase immersion the fluid boils on hot surfaces and condenses on a coil above the bath, exploiting latent heat for very high flux with little or no pumping.
Immersion captures essentially all of the heat, deletes every fan, and tolerates layouts that would be thermally impossible in air. The costs are operational rather than thermal: tanks are heavy enough to matter for floor loading, servicing means hoisting a dripping sled out of a bath, and not every component is compatible — some thermal interface materials, labels, optics, and mechanical drives object to submersion. Two-phase also faces regulatory and supply pressure on the fluorinated fluids it has relied on, which is a procurement risk more than a technical one.
The loop behind the rack — CDU, approach temperature, warm water
Whatever touches the silicon, the heat still has to reach the sky. Liquid halls split that path: the secondary loop is the clean, tightly controlled circuit running into the racks; the primary loop is facility water heading for chillers, dry coolers, or a tower. Between them sits a CDU — heat exchanger plus pumps, filtration, and instrumentation. Its job is to isolate, not merely to pump: known-good fluid at a known pressure in front of expensive hardware. The number that sizes it is approach temperature, how many degrees warmer the secondary supply ends up than the primary water feeding the exchanger.
The counterintuitive goal is then to run that loop as warm as the hardware allows; industry guidance indexes liquid-cooling classes by maximum facility supply temperature for exactly this reason. If the loop is happy above ambient most of the year, heat rejects through dry coolers or a tower without mechanical refrigeration and the chiller becomes a trim device. That deletes the building's largest non-IT consumer for most operating hours — the real efficiency argument for liquid.
Water quality, materials, and leak risk
Liquid cooling fails through chemistry far more often than through thermal shortfall. The secondary loop is not tap water: it is treated fluid — typically deionised water with corrosion inhibitors, sometimes a glycol blend where freeze protection is needed — plus biocide, filtration, and a maintenance schedule. Cold-plate channels are narrow, so particulate that would be invisible in a building loop can foul them, and biological growth in a warm, dark, wet circuit is a realistic failure mode.
Materials compatibility is the other half: mixing metals invites galvanic corrosion, and elastomers, gaskets, and quick-disconnect seals must be qualified against the fluid. Then the fear everyone names first — leaks. The mitigations are unglamorous and effective: dripless blind-mate couplings, leak-detection cabling under manifolds, interlocks that isolate a rack automatically, negative-pressure designs that pull air in rather than push fluid out. Liquid mostly converts a thermal risk into a plumbing one.
Heat reuse, PUE, and what the metrics flatter
Warm return water makes heat reuse conceivable, and the constraint is grade, not quantity. Return water arrives well below what most district-heating networks want, so reuse needs either a customer who takes low-grade heat directly or a heat pump to lift it, which consumes energy and shifts the economics. Reuse is a siting and offtake decision made years ahead of the hall, not a retrofit.
PUE deserves a warning. It is facility energy over IT energy, and liquid improves it genuinely by deleting chiller and CRAH load. But part of the gain is bookkeeping: server fans counted as IT load, inflating the denominator, so removing them makes the ratio look worse even as total energy falls. Where CDU pumps are metered moves it too. Judge designs on total energy per unit of useful work, and watch water usage effectiveness alongside — evaporative rejection buys a flattering PUE by spending water.
Retrofit versus greenfield
The same density lands very differently depending on whether you own the building. A retrofit is bounded by things you cannot renegotiate: structural floor loading that may rule out immersion tanks, no piping path to the rack rows, a chiller plant sized for an air-cooled hall, ceiling height, and a live tenant load you cannot switch off. That is precisely the niche rear-door exchangers occupy — the highest density you can buy without opening the servers or the slab.
A greenfield build chooses the loop temperature first and designs everything downstream of it: dry coolers instead of chillers, manifolds designed in, CDUs placed with service clearance, fewer and denser racks, floor structure specified for the intended tier. Fix target density and supply-water temperature before anything else, let the hydraulics fall out of those two, and leave headroom — the one reliable trend is that the next generation asks for more.