768GB of cheap Intel Optane DIMM memory sticks used to run 1-trillion-parameter LLM on a system with a single GPU — local Kimi K2.5 install achieved roughly 4 tokens per second
Redditor found 768GB of affordable Optane sticks second-hand.
A Redditor has caused a stir by coaxing a workstation build using Optane PMem DIMMs as RAM to run a 1-trillion-parameter LLM. APFrisco explains in a mini tutorial/guide on the Local LLaMA subreddit how they bought some used Intel Optane Persistent Memory, acquired relatively cheaply second-hand, to “run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second” on their Xeon workstation.
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/secfrom r/LocalLLaMA
Central to the headlining feat was the Redditor’s sourcing of six Optane PMem (DCPMM) sticks. The discontinued memory format was designed to bridge the DRAM-SSD divide. While the 768GB of Optane (6x 128GB) does indeed offer far lower latency than the best NVMe SSDs, it is still two or three times slower than DRAM. These characteristics are still rather sweet for LLM inference frameworks, and the second-hand price was “much less than what the equivalent DRAM capacity would cost.” But, alas, Optane is dead, so this is an exotic solution.
APFrisco’s hardware specs were given as follows:
- Intel Xeon Gold 6246 CPU
- Tyan S5630GMRE-CGN motherboard
- Asus Dual GeForce RTX 3060 OC 12GB GPU
- 6x 32GB Samsung 2666MHz DDR4 ECC DRAM sticks
- 6x 128GB Intel Optane DCPMM PC4-2666 NMA1XBD128GQS persistent memory modules
- Western Digital WD SN850X 2TB M.2 2280 NVMe SSD
- ASRock Steel Legend SL-850G 850W 80 PLUS GOLD & Cybenetics Platinum Fully Modular Power Supply
- Silverstone SST-GD08B (Black) Grandia Series Home Theater PC Case
The build was configured with the Optane in memory mode and the Samsung DDR4 as cache.
The software side of the equation relied on the aforementioned Kimi K2.5’s mixture-of-experts architecture. APFrisco used a hybrid GPU/CPU inference methodology with llama.cpp. Also, to optimize processing, the routing components were shoehorned into the 12GB GPU using llama.cpp’s 'override-tensor' flag.
The Redditor is rather proud of the resulting ~4 tokens per second performance. “Given the fact that this is a trillion-parameter frontier-class model running on such a limited hardware budget, I would consider it to be a great success,” writes APFrisco. They go on to lament Intel’s withdrawal of Optane products.
If you are interested in this rig rundown and what it achieved in terms of local LLM inference, you can find some more details about the configuration in the source post. Furthermore, APFrisco sticks around in the comments to answer questions. They also appear to benefit from recommendations about how to achieve even better results, given the foundation they have laid.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
The bigger picture, though, seems to be that there is room for a memory product in the chasm between DRAM and SSDs, particularly for LLMs. Many expect that the gap will soon be bridged by the CXL (Compute Express Link) standard, which promises huge pools of affordable, byte‑addressable memory for these kinds of workloads.
Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.
Mark Tyson is a news editor at Tom's Hardware. He enjoys covering the full breadth of PC tech; from business and semiconductor design to products approaching the edge of reason.
-
bit_user Reply
Nah, the key detail is that they were purchased used. When new, I think the GB/$ wasn't that much better than DRAM. However, DRAM is now much more expensive, which makes alternative solutions like Optane DIMMs much more attractive.usertests said:Intel's poor timing continues to impress years after the fact.
You really can't use this example to reason about Optane's market viability. -
JTWrenn Reply
Agreed, the main reason being that fab usage would be about the same for both so it's not like we would get cheaper anything out of it. Optane just like SSD's would be more expensive anyway. Optane was cool tech that was just not cheap enough to make sense.bit_user said:Nah, the key detail is that they were purchased used. When new, I think the GB/$ wasn't that much better than DRAM. However, DRAM is now much more expensive, which makes alternative solutions like Optane DIMMs much more attractive.
You really can't use this example to reason about Optane's market viability. -
usertests Reply
The $/GB wasn't dramatically less than DRAM, but that could have improved as production scaled up. But they were losing money and didn't stay the course.bit_user said:Nah, the key detail is that they were purchased used. When new, I think the GB/$ wasn't that much better than DRAM. However, DRAM is now much more expensive, which makes alternative solutions like Optane DIMMs much more attractive.
You really can't use this example to reason about Optane's market viability.
Since it's off the table, we see solutions like High Bandwidth Flash stepping into the empty tier instead. I guess you can disregard write endurance if the model isn't changing rapidly. -
bit_user Reply
Not faster than DRAM, unless they managed to solve the issues that kept it from scaling in the Z-dimension like it was meant to.usertests said:The $/GB wasn't dramatically less than DRAM, but that could have improved as production scaled up.
HBF has yet to be deployed in any solution, so it has yet to be a proven alternative.usertests said:Since it's off the table, we see solutions like High Bandwidth Flash stepping into the empty tier instead.
HBF should have a structural price advantage over Optane, because it's true 3D NAND. -
hwertz i looked into pdimms before. Interesting, because the PERSISTENT part of them worked on very few chipsets. they're so cheap because that part won't work on most ssystems so most potential buyers take that to mean they won't work at all. But set to run without persistence, they just show up as somewhat slow DIMMS and work in far more machines, which is what it sounds like is being done here.Reply