Nvidia's memory costs soar 485%, latest AI systems now cost $7.8 million to build — memory now comprises 25% of the total cost, Rubin GPUs a mere $50,000 apiece

News
By published

$2 million for memory alone.

Nvidia
(Image credit: Nvidia)

As prices of components are increasing rapidly due to high demand from the AI sector, the cost of these machines is also increasing significantly. Morgan Stanley Research estimates that a next-generation Vera Rubin-based VR200 NVL72 rack will cost major hyperscale cloud service providers (CSPs) around $7.8 million per unit (via @Aaronwei3n), which is tangibly more than about $4 million per GB300 NVL72. Furthermore, because every VR200 NVL72 rack packs plenty of DRAM and NAND, memory now accounts for around 25% of the total cost.

Nvidia plans to charge $55,000 per Rubin GPU and $5,000 per Vera CPU when selling them in volume inside VR200 NVL72 chassis to hyperscalers, according to Morgan Stanley. Although the upcoming VR200 NVL72 racks use the already familiar Oberon chassis, they use more sophisticated switching, networking, printed circuit board (PCB), cooling, power supply, and even chip packaging components, which increases bill-of-material (BOM) costs and eventually the price of the systems. As a result, each VR200 NVL72 will cost hyperscalers around $7.8 million, according to Morgan Stanley, which is higher than around $7 million we were told by one of our sources in late March. Meanwhile, the cost of memory within a VR200 NVL72 rack will be about $2 million, up 435% from the memory cost in GB300 NVL72, according to the same figures.

There are several reasons why the cost of memory is expected to account for 25% of the cost of a VR200 NVL72 system and why the system carries $2 million worth of memory.

Latest Videos From

First up, each of such racks now contains 54 TB of LPDDR5X memory, up from 17 TB of LPDDR5X in the case of a GB200 NVL72, a threefold increase. SemiAnalysis estimates that Nvidia paid $8 per GB per GB of LPDDR5X in Q1, though that price may increase as demand rises in the coming quarters, especially if we are talking about SOCAMM2 modules that are expensive to make and test. In any case, even at $8 per GB, each GB200 NVL72 machine carries $136,000 worth of LPDDR5X memory, whereas each VR200 NVL72 system will contain $408,000 worth of LPDDR5X content. If the price rises to $10, we are talking about $540,000 for LPDDR5X alone. Note that even $10 per GB may be an underestimate* as Nvidia adds its own markup.

Secondly, each VR200 NVL72 rack carries about $1 million or more of 3D NAND storage, up from virtually zero inside GB200 NVL72.

As a result, $2 million of memory content per Vera Rubin NVL72 rack is not something completely unexpected: the system uses a lot of LPDDR5X and 3D NAND memory (not to mention HBM4 memory onboard of Rubin GPUs), and memory now comes at massive prices.

*Contract price of DDR5 memory is now between $12 and $16 per GB, depending on various factors and luck, according to Framework. Spot price for DDR5 was about $20 per GB on average at press time, according to DRAMeXchange. LPDDR5X is more expensive than DDR5. When installed on SOCAMM2 modules (which are exclusively used by Nvidia's Vera CPUs), it will get even more costly, especially when Nvidia's markup is added.

Google Preferred Source

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.

TOPICS
Anton Shilov
Anton Shilov
Contributing Writer

Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.

8 Comments Comment from the forums
  • ekio
    Some people will be made fun of in the future, regarding how much they were stupidly paying for compute, allowing 90 percent margins for the mob, I mean from Ngreedia.
    Reply
  • alan.campbell99
    Data centre endeavours are already debt-laden as it is and this will just add more to the fire, VC/PE money is not an infinite resource. I don't like this timeline.
    Reply
  • JamesJones44
    alan.campbell99 said:
    Data centre endeavours are already debt-laden as it is and this will just add more to the fire, VC/PE money is not an infinite resource. I don't like this timeline.
    Something to watch is a lot of the memory producers have resisted adding capacity. I think if they thought this was going to last they would bring more production online. It will be interesting to see where we are a year from now.
    Reply
  • timsSOFTWARE
    There haven't been any major technological breakthroughs since 2017, with the invention of the transformer, and the "attention is all you need" paper. Everything that has happened since then has been sort of low-value. Even though there are companies that will IPO with trillion-dollar valuations, not having contributed all that much scientifically.
    Reply
  • usertests
    timsSOFTWARE said:
    There haven't been any major technological breakthroughs since 2017, with the invention of the transformer, and the "attention is all you need" paper.
    AI says mRNA vaccines and quantum computers (debatable, quantum supremacy when?). I would add drones, specifically the use of low-cost drones in warfare, which have become completely ubiquitous.

    In computer hardware, most of the action has been in 2.5D/3D packaging, such as the giant interposers being used (mostly for AI chips). GAAFETs and backside power delivery are major on the transistor side.

    LLMs as we know them were completely worthless in 2017. Now AI-assisted coding is here to stay and they are digging up security vulnerabilities at an alarming rate (also debatable, maybe a pretext for regulating competition out of existence).
    Reply
  • Shiznizzle
    If and when the chinese bring their fabs online will they keep the prices just as high? My guess is they will.

    I am ok with going outside again and sniffing grass on my bikes just like when i was a kid. What i am not ok with, as are many others it seems, is paying those prices for tech.

    Us, joe public, not buying expensice stuff will have an effect.

    What i find interesting is that the demand is not there but everybody is keeping prices sky high.
    Reply
  • usertests
    Shiznizzle said:
    I am ok with going outside again and sniffing grass on my bikes just like when i was a kid. What i am not ok with, as are many others it seems, is paying those prices for tech.
    You’re Breathing Plastic: Study Finds 4% of City Air Pollution Is Microplastics
    Air pollution may be changing people’s bodies, adding fat while draining lean mass
    Air pollution directly linked to Alzheimer’s risk, scientists say
    LORD LET THIS MAN SNIFFIZZLE POLLUTION AND PAY FOR TECH
    Reply
  • timsSOFTWARE
    usertests said:
    AI says mRNA vaccines and quantum computers (debatable, quantum supremacy when?). I would add drones, specifically the use of low-cost drones in warfare, which have become completely ubiquitous.

    In computer hardware, most of the action has been in 2.5D/3D packaging, such as the giant interposers being used (mostly for AI chips). GAAFETs and backside power delivery are major on the transistor side.

    LLMs as we know them were completely worthless in 2017. Now AI-assisted coding is here to stay and they are digging up security vulnerabilities at an alarming rate (also debatable, maybe a pretext for regulating competition out of existence).

    Sorry - I was talking about the field of AI/LLMs specifically. They've gotten bigger, and techniques like RLHF have been employed to improve them incrementally, but they are all still based on the transformer architecture from the 2017 "attention is all you need" paper.

    AI assistance for coding - like other forms of writing assistance - is very broad/generic. For the companies using AI to churn out massive volumes of code quickly, and deploying that code in production - I don't think the last chapter of that book has been written yet. I don't think that's going to work out for those companies in the end; it's just too hard to maintain and keep a reasonable level of quality, as expected for commercial products.

    But it has replaced Stack Overflow, etc., as the first place to go when you have a question, can accelerate learning new things by a lot, and can be highly effective when given smaller, well-defined/sandboxed tasks, where the lack of long-term memory/context window limits aren't an issue.
    Reply