AMD's fastest AI chips could get a memory boost as competition with Nvidia intensifies — Instinct MI300 will soon come with new memory configurations

Recently, Nvidia launched its H200 with 141GB of HBM3E and stole quite a lot of thunder from AMD; Nvidia's latest came just after AMD had released its Instinct MI300A with 128GB of HBM3 memory and the Instinct MI300X AI accelerators with 192GB of HBM3 memory. However, AMD may be prepping new versions of Instinct MI300-series products with more memory. 

Mark Papermaster, chief technology officer of AMD, implied this possibility during his presentation at the Arete Investor Webinar Conference last week.

"We are not standing still," said Papermaster (via SeekingAlpha). "We made adjustments to accelerate our roadmap with both memory configurations around the MI300 family, derivatives of MI300, the generation next. […] So, we have 8-Hi stacks. We architected for 12-Hi stacks. We are shipping with MI300 HBM3. We have architected for HBM3E." 

It should be noted that AMD's Instinct MI300A is equipped with 128GB of HBM3 memory using 8-Hi known good stack die (KGSD), whereas the Instinct MI300X comes with 192GB of memory and uses 12-Hi KGDSs. For now, none of AMD's products use HBM3E, but it looks like the company could use it for some of the future versions of AMD's Instinct MI300-series products.

By using HBM3E instead of HBM3, AMD can significantly increase the memory bandwidth available to its AI and HPC GPU or APU. Changing memory devices attached to its processors is a fairly easy thing to do, so expect AMD to use HBM3E rather sooner than later. 

According to AMD's performance numbers, the Instinct MI300X outperforms Nvidia's H100 80GB GPU, which is widely used by major hyperscalers such as Google, Meta (Facebook), and Microsoft. The Instinct MI300X is also likely to be a strong contender against Nvidia's upcoming H200 141GB GPU. Meanwhile, it remains to be seen what to expect from AMD's hypothetical Instinct MI300-series with HBM3E memory.

"We have been very closely working with our customers, and what we can tell you this, I willl tell you right now: that race has begun, and it is going to be a competitive race," said Papermaster. "You are going to see that back and forth like you have always seen when you have competition. It is going to be great for the market. It is certainly spurring us at AMD to be at our very best of innovation. […] Stay tuned with us as we will share more details forthcoming on that roadmap because it is indeed a multiyear roadmap that we have laid out."