Meta's Llama 4 trains on over 100,000 Nvidia H100 GPUs

Meta has recently released a new update about Llama 4, its upcoming large language model. In a recent earnings call, CEO Mark Zuckerberg shared that they’re training the Llama 4 models on a cluster consisting over 100,000 Nvidia H100 GPUs—bigger than anything he's seen reported for what others are doing.

This new large language model, expected to come out in early 2025, will first roll out its smaller versions. While there aren’t many details on its specific abilities, Zuckerberg hinted that Llama 4 will have new features, better reasoning skills, and overall faster.

Meta’s approach with Llama 4 keeps up its pattern of making models free to download, unlike OpenAI and Google, which limit access through APIs only. This model is especially popular with startups and researchers wanting more freedom over how they use AI.

With such a massive computing setup, the energy demands are equally large, estimated at 150 megawatts—five times the energy demand of the largest U.S. national lab supercomputer. Meta’s budget reflects this scale, with infrastructure spending expected to reach $40 billion in 2024, a 42 percent jump from 2023.

Even with these significant costs, Meta’s finances are solid, showing a 22 percent increase in sales largely thanks to ad revenue, which has helped balance out a 9 percent rise in operating costs.