Microsoft Maia 300 AI chip arrives with monstrous 288 GB HBM4 memory and 18 TB/s bandwidth

Microsoft has officially unveiled the Maia 300, its latest custom-designed AI accelerator intended to reduce the company's dependence on Nvidia and AMD hardware. It arrives shortly after OpenAI debuted its in-house silicon, Jalapeno.  The chip succeeds the Maia 200, which launched earlier this year. The first Maia chip, the Maia 100, surfaced in 2023. 

The Maia 300 uses a multi-chiplet design comprising a main compute section manufactured using TSMC's advanced N2P (2 nm) process. The package also incorporates separate memory and I/O chiplets, allowing Microsoft to manufacture different components using processes suited to their respective requirements. According to Reuters, some of the supporting silicon uses a less expensive 3 nm manufacturing process.

The accelerator delivers up to 13.8 PFLOPS of FP4 compute performance and 6.9 PFLOPS at FP8 precision. Its SoC power rating stands at 950 W, substantially higher than the Maia 200's 750 W rating. Microsoft has equipped the Maia 300 with eight HBM4 stacks, offering a combined 288 GB of memory and 18 TB/s of bandwidth. Each stack contributes 36 GB of capacity, with the HBM4 memory operating at 9 Gb/s per pin. For comparison, the Maia 200 features 216 GB of HBM3E memory with 7 TB/s of bandwidth, making the new accelerator useful for memory-intensive inference workloads.

Microsoft has also upgraded the accelerator's interconnect capabilities. The Maia 300 delivers 3.2 TB/s of bidirectional scale-up bandwidth and uses a two-tier Ethernet-based network with a custom AI transport protocol. A single rack accommodates 96 Maia 300 accelerators, while the scale-up architecture supports clusters containing approximately 12,000 accelerators.

Beyond its raw specifications, Microsoft claims Maia 300 can deliver 40% more tokens per megawatt than leading GPU-based AI systems. A fully populated 96-accelerator configuration reportedly offers up to seven times the rack-scale token throughput of Maia 200. However, the company has not provided sufficient details about the competing GPU configurations to independently establish the extent of its efficiency advantage.

Microsoft intends to begin deploying Maia 300 in production environments in 2027, with plans to eventually scale the infrastructure to multiple gigawatts of capacity. Reuters reports that the company could deploy more than one million Maia 300 chips next year, highlighting its ambitions to become less dependent on third-party AI accelerators.