Meta details Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio
- Meta's Prometheus supercluster in New Albany, Ohio already has hundreds of thousands of GPUs across multiple buildings, and the first phase trained the Muse Wonder models, with a goal of roughly 1 million GPUs at full scale.
- Training runs reached more than 85% goodput for pre-training and more than 90% for reinforcement learning, and mean time between interruptions rose from about 47k GPU hours to more than 54K after firmware fixes.
- Prometheus runs two NVIDIA platforms, and the Catalina GB200 NVL72 (Ariel variant) pairs each Blackwell GPU with a Grace CPU and 2x ConnectX-7 400G NICs to deliver 800G of scale-out per GPU, with a Vera Rubin NVL72 platform arriving soon.
- The article says Prometheus uses closed-loop liquid cooling to operate efficiently but does not give figures for it.
Hacker News opinions
Posting this two days before earnings usually means the capex numbers are going to look ugly.
Cloud providers used to pay cash for hardware. Now they borrow at 8% while RAM and storage costs have jumped, so it's going to get rough out there.
Meta's crawler at 57.141.0.0/24 made about 90% of requests to my server last week and kept hitting it every 2 seconds even after I returned 403s.
If you want to drop it at the kernel, an nftables set with the interval flag and a reject rule on tcp 80 and 443 does the job.
I don't think it's incompetence, I think it's by design.
I blocked Meta, OpenAI and Anthropic and they still find ways back in, so I just put everything behind Cloudflare. Least bad option for us.
Hot spares make sense when you push GPUs this hard. The job survives until someone swaps the bad part.
If the 1:1 CPU-to-GPU ratio was just for I/O, why not ConnectX-8? Other hyperscalers shipped NVL72 with it around the same time at the same 800Gb/s. Edit: I checked and GB200 NVL72 never shipped with CX-8, so I was wrong there. Compute availability and timing probably drove it.