Microsoft debuts custom silicon for AI and cloud: 128-core ARM Cobalt 100 and 5 nm Maia 100 aim to power next gen AI training and inference needs

Speculation about Microsoft's plans to take the custom silicon route has been running rife for quite some time now. Microsoft has been using Qualcomm-based processors such as the SQ1, SQ2, and the SQ3 (Snapdragon 8cx Gen 3) for its Surface lineup. However, reports about the company mulling its own in-house ARM designs for server started cropping up back in 2020 itself.

At the ongoing Microsoft Ignite conference, the company unveiled two custom silicon IPs designed to cater to its ambitious AI plans — the Azure Maia 100 AI accelerator and the Azure Cobalt 100 custom CPU. Microsoft will first deploy these chips early next year to its own datacenters that power Copilot and the Azure OpenAI service.

Microsoft isn't revealing much technical info about the chips themselves, though. It does, however, note that the Maia 100 is a 5 nm chip sporting 105 billion transistors, making it one of the largest such chips to be manufactured on this process. Maia 100 will run cloud-based training and inferencing for OpenAI models and ChatGPT, and has been specifically designed to train OpenAI's large language models.

Maia 100 will be fully liquid cooled, and Microsoft has even designed custom racks with liquid cooling radiator "sidekicks" to house these chips.

The Cobalt 100, on the other hand, is a 64-bit 128-core CPU that is designed for general workloads in the Microsoft Cloud including Teams and SQL. Microsoft says that Cobalt 100 is 40% more powerful over the current crop of Azure ARM chips.

In addition to Maia 100 and Cobalt 100, Microsoft also announced the public preview of Azure NC H100 v5 virtual machines (VMs) for running AI and high-performance computing (HPC) workloads. These VMs are based on the Nvidia Hopper H100 GPUs and AMD EPYC Genoa processors. 

Customers can choose from two NC H100 v5 configs:

• 40 vCPUs, 320 GB RAM, 1 H100 GPU with 94 GB HBM3, and 40 GBps Azure network bandwidth
• 80 vCPUs, 640 GB RAM, 2 H100 GPUs with 188 GB HBM3, and 80 GBps Azure network bandwidth

Microsoft is not leaving out AMD, though. The Redmond-giant has also announced it is now offering VMs featuring AMD's latest Instinct MI300X GPU with a 1.5 TB HBM3 per VM. An ND MI300X v5 VM leverages eight MI300X cards interconnected by Infinity Fabric 3.0 with 192 GB of HBM3 memory per GPU that operates at 5.2 TB/s speeds. According to Microsoft, specifications of the ND MI300X v5 VM include:

• 400 Gb/s NVIDIA Quantum-2 CX7 InfiniBand per GPU with 3.2 Tb/s per VM
• 4th Gen Intel Xeon Scalable processors
• PCIE Gen5 host-to-GPU interconnect with 64 GB/s bandwidth per GPU
• 16 Channels of DDR5 DIMMs

Microsoft's custom silicon is currently geared towards the server side of things. It is not clear whether the company would bring some variant of Cobalt 100 to its Surface linup or continue to rely on Qualcomm, and possibly MediaTek, offerings — not necessarily a bad idea given the impressive early numbers posted by the Snapdragon X Elite.

Buy the Microsoft Surface Laptop Studio 2 with RTX 4050 on Amazon