On October 19, 2026, four Meta infrastructure leaders described Prometheus, the supercluster in New Albany, Ohio, whose first phase trained the Muse models, including Muse Wonder. It now holds "hundreds of thousands of GPUs across multiple interconnected buildings" and is meant to reach roughly one million. It currently runs two NVIDIA platforms, with a third on the way. Catalina is GB200 NVL72 in a two-rack "Ariel" layout that pairs every GPU with a Grace CPU and two 400G NICs, giving 800G of scale-out per GPU early. Clemente is GB300 NVL72, with 288 GB of HBM per GPU, in a single rack. The third, San Miguel, is Vera Rubin NVL72, which Meta expects to give 10-20% better performance per watt than GB300.
The rack is the unit, and eight GPUs wait
Moving from 8-GPU NVLink domains to 72 increases "the fault blast radius by 9X," the authors write. Meta's answer: "each NVL72 domain runs a 64-GPU slice and keeps its remaining eight GPUs as in-rack spares," and about 2% extra racks are held for whole-rack faults. Their worked example: a 32,768-GPU training run needs 523 racks, or 37,656 GPUs, "about 13% above the active footprint," most of it from "the 64-of-72 slice geometry rather than being an absolute reliability tax." By our arithmetic, 37,656 is 14.9% more than 32,768; the 13% figure is the share of provisioned GPUs that are not active.
From 1,000 interruptions a day to a handful
In mid-2025, "interruption rates reached a peak of nearly 1,000 events per 8K GPUs daily", which works out to one interruption roughly every 200 GPU-hours. A year later Meta reports a 99% reduction. During the Muse Wonder runs the mean time between interruptions (MTBI) was about 47,000 GPU-hours, with more than 85% goodput (the share of time spent on useful training) for pre-training and more than 90% for reinforcement learning. Firmware fixes have since pushed MTBI past 54,000 GPU-hours. The target is "two interruptions per 8K GPUs per day, which is ~100k GPU hours MTBI." The main failure classes were:
- NVLink signal integrity across more than 600,000 NVLink connections, now screened with diagnostic burn-in before production.
- HBM and LPDDR memory faults on GB300, handled with firmware settings, memory scrubbing and predictive retirement of failing devices.
- Firmware rollout speed: Meta can now push fixes to more than 100,000 GPUs "within a few hours", and the P90 time to upgrade one pod fell from over seven hours to two.
- Power: synchronized training causes load swings of 30-40%, and starting or pausing a job "can shift loads of more than tens of megawatts in just seconds." A power-capping system called Dynamo Dimmer, rack batteries, capacitor trays and site-level storage absorb the swings.
The authors describe a "co-engineering model with NVIDIA that goes beyond a traditional vendor dynamic", with joint triage of every halt. NVIDIA owns firmware, signal diagnostics and memory handling.
What remains uncertain. The post gives no exact GPU count, cost or site power figure, and it does not define how goodput is measured. The 99% reduction is measured from the worst point.
Analysis: The planning lesson applies well beyond Meta. Anyone renting or buying NVL72 capacity is paying for idle spares, whether they appear on the invoice or are built into the hourly price. MTBI sets how a training stack must behave: at 54,000 GPU-hours, a 4,096-GPU job is interrupted about every 13 hours by our arithmetic. Jobs that cannot drop a failed data-parallel replica and keep going, or that checkpoint slowly, lose that time every half day. Power transients are now partly a scheduler's problem too.
Meta's October 19 Prometheus post says interruptions on its NVL72 supercluster peaked at nearly 1,000 per 8K GPUs a day in mid-2025 and have fallen 99%, with MTBI now above 54,000 GPU-hours against a ~100,000 target; each 72-GPU rack runs 64 GPUs and holds 8 as spares, so a 32,768-GPU run provisions 37,656 GPUs, and NVIDIA co-owns firmware, signal and memory fixes.