Microsoft Reveals Custom 128-Core Arm Datacenter CPU, Massive Maia 100 GPU Designed for AI

Microsoft's AI-first initiatives have brought it to custom chips. At Microsoft Ignite, the company's conference for partner developers, Microsoft announced two custom chips designed in its Redmon, Washington silicon lab: the Azure Maia 100 AI Accelerator and Azure Cobalt 100 CPU. Both chips are being produced on "the latest process node" at TSMC, Microsoft told Tom's Hardware.

These chips will show up "early next year" in Microsoft's data centers, which will start by running Microsoft's Copilot, Azure OpenAI and other services. When asked how many chips will be available, a spokesperson only said that Microsoft is in the "first phases of deployment" with "initial servers" online in its data centers. The company will still offer services to customers using chips from partners like Nvidia and AMD.

Microsoft isn't releasing much in the way of spec details, making it difficult to compare the new offerings to Nvidia and AMD options.

When asked for more details, Microsoft told Tom's Hardware that Maia 100 is manufactured on a 5-nanometer process and boasts 105 billion transistors, "making it one of the largest chips that can be made with current technology."

"Maia’s servers are designed with a fully custom Ethernet-based network protocol with aggregate bandwidth of 4.8 terabits per accelerator to enable better scaling and end-to-end workload performance," Microsoft added.

What we do know is that the Maia 100 AI Accelerator - a GPU - is designed for large language models. Microsoft says that OpenAI "provided feedback" about the Maia 100 designs.

“We were excited when Microsoft first shared their designs for the Maia chip, and we’ve worked together to refine and test it with our models," OpenAI CEO Sam Altman stated in Microsoft's blog post. "Azure’s end-to-end AI architecture, now optimized down to the silicon with Maia, paves the way for training more capable models and making those models cheaper for our customers.”

Brian Harry, a Microsoft technical fellow heading up the Maia team, said that Maia 100 was made specifically for Microsoft's existing infrastructure, and that it can "yield huge gains in performance and efficiency."

Meanwhile, the Cobalt 100 CPU is Arm-based, which Wes McCullough, corporate vice president of hardware product development said in the blog post, "is designed with power efficiency in mind."

"Azure Cobalt uses a licensed design from Arm based on ARM Neoverse CSS that is customized for Microsoft’s purposes," Microsoft told Tom's Hardware. "The 64-bit 128-core chip represents performance improvements of up to 40% over current generations of Azure Arm servers."

The company also states that designing its own chips allows it to increase cooling in data centers and to boost the capacity in existing facilities. The company said that there were no server racks that could "house the unique requirements of the Maia 100 server boards," which required building racks from scratch.

"These racks are wider than what typically sits in the company’s datacenters," Microsoft's blog post reads. "That expanded design provides ample space for both power and networking cables, essential for the unique demands of AI workloads."

The company also needed to come up with a liquid cooling solution to chill these chips during complex AI tasks, but didn't have data centers set up for "large liquid chillers." So Microsoft is using what it calls a "sidekick" that can sit next to a rack of Maia 100 servers, send cold liquid to cold plates attached to the Maia 100 chips, and then send the warm liquid back to the sidekick to remove the heat and return the liquid to the rack again.

Microsoft says it has shared the custom rack and cooling designs with industry partners.

"All the things we build, whether infrastructure or software or firmware, we can leverage whether we deploy our chips or those from our industry partners,” said Pat Stemen, partner program manager on Microsoft's Azure Hardware Systems and Infrastructure team. “This is a choice the customer gets to make, and we’re trying to provide the best set of options for them, whether it’s for performance or cost or any other dimension they care about."

The company is already working on second-generation versions of both the Maia AI Accelerator and Azure Cobalt CPU. 

In addition to the new hardware, Microsoft also announced Azure Boost, which it says can make storage and networking processes quicker by moving them from host servers to hardware and software designed specifically for those purposes.

Microsoft also announced further partnerships with both AMD and Nvidia. On the Team Red side, Microsoft will add AMD MI300X virtual machines to its Azure platform. For Ndvidia, there's a preview of the new NC H100 v5 Virtual Machine Series designed for Nvidia's H00 Tensor Core GPUs.

Many details about Maia and Cobalt were not available when this was written. But we have reached out to Microsoft with further questions, so expect more coverage here soon. But one thing is for certain: In a crowded landscape of bespoke, cloud-focused data center CPUs with players like Meta, Oracle, and Google, this move from Microsoft is likely to have a massive impact. Old-school data center mainstays like Intel, AMD, and Nvidia, as well as comparative newcomers like Ampere, are likely to be impacted by these announcements and will have to make bold moves of their own.