Smart Modular Brings Optane Memory to AMD EPYC Servers
Kestral PCIe card uses an Intel FPGA to interface with AMD, Arm systems.
One of the advantages that servers based on Intel's 2nd and 3rd Generation Xeon Scalable processors have over AMD EPYC-powered machines is support for Optane Persistent Memory modules that can be used to expand main memory capacity relatively cheaply--something that is important for many applications. But Smart Modular has developed a card that can add Optane PMMs to any machine. Ironically, the card is Intel-based.
Apparently, demand for storage-class memory on existing AMD and Arm machines is significant enough to encourage Smart Modular to develop its special Kestral PCIe Optane add-in-card for these servers. The Kestral PCIe Optane AIC carries up to four 512GB Intel Optane Persistent Memory modules and can be plugged into any PCIe 3.0/4.0 x16 slot to add up to 2TB of Optane memory. A PCIe Gen4 x16 slot supports a peak bandwidth of 32GBps in each direction, which is higher when compared to one DDR4-3200 memory channel (25.6 GBps), but is lower when compared four DDR4 channels (102.4 GBps).
Intel's Xeon Scalable 'Cascade Lake' and 'Ice Lake-SP' processors communicate with Optane Persistent Memory modules that are plugged into regular DDR4 memory slots using the company's proprietary DDR-T protocol, which is not supported by competing CPUs. The protocol is required for Optane PMMs to provide a 64B cache line access granularity while taking into account the nature of these modules (which use a special Optane memory controller).
To make Intel's Persistent Memory modules work on its Kestral PCIe AIC, Smart Modular carries Intel's Stratix 10 DX FPGA, which fully supports the DDR-T protocol and works like a DDR4-to-DDR-T memory translator hub. Also, the AIC can use FPGA's four Arm Cortex-A53 cores to accelerate/enable certain workloads. The board comes with 2GB of DDR4 ECC memory and an 8GB eMMC storage module. The card consumes less than 75W in memory expansion use cases and less than 100W when used for memory acceleration. It comes in a full-height half-length (FHHL) form-factor and is equipped with a dual-slot passive cooler. The board can be used for field upgrades of existing servers to support new workloads.
While the card is compatible with virtually all existing servers (and even future machines with Nvidia Grace CPUs), perhaps the biggest challenge to make it work efficiently will be making software use it properly. Perhaps, for those customers who already use thousands of AMD EPYC-based machines and who need to repurpose them for different workloads that require expensive memory expansions, usage of Kestral with Optane Persistent Memory modules will make sense. Others may opt to expand their memory capacity using regular DIMMs, or just wait for next-generation servers with CXL protocol support.
"With the advancement of new interconnect standards such as CXL and OpenCAPI, Smart's new family of SMART Kestral AICs addresses the industry's need for a variety of new memory module form factors and interfaces for memory expansion and acceleration," said Mike Rubino, SMART Modular’s vice president of engineering.
Smart Modular's Kestral PCIe Optane AIC is now available for orders from the company. Pricing is unknown, but don't expect it to be low as we are dealing with a highly-specialized product based on a high-end FPGA.
Stay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.
-
abufrejoval Cool!Reply
Very niche, but cool nevertheless.
I've been very disappointed when Martin Fink's 2014 projections about the memristor replacing everything from DRAM to tape wound up nothing but hot air and a split HP.
But to see that Intel's attempt to the make runner-up phase change materials exclusive to Intel has finally failed, is actually a bit sweet.
I thought production of Optane had already stopped with both Micron and Intel discarding their bits, but for a niche so small, enough stockpiles may be left.
If someone doesn't know where to send a leftover, I'd be volunteering to take two or three!
They'll be a quite natural replacement for the FusionIO boards I'm recycling from a previous life in a lab. -
TerryLaze
You still need an intel system to make it show up as ram, this is just ram on a pci card which is nothing new, it only shows up as storage, which is something compared to nothing at all but it doesn't give you all the major benefits.abufrejoval said:But to see that Intel's attempt to the make runner-up phase change materials exclusive to Intel has finally failed, is actually a bit sweet. -
abufrejoval
Well if you want to take advantage of its non-volatility, you may have to tailor your software anyway. And then from what I've learned nobody would mistake Optane for RAM in any case, because the performance is significantly slower, even if it sits on the memory bus.TerryLaze said:You still need an intel system to make it show up as ram, this is just ram on a pci card which is nothing new, it only shows up as storage, which is something compared to nothing at all but it doesn't give you all the major benefits.
And if you in fact you do (mistake it for RAM), there is a bit of a chance you'll block off an entire memory channel (with lots of RAM-DIMMs) waiting for NV-DIMM requests to release it. Storage has been accessed asynchronously "forever", even virtual memory access won't tie a memory channel on page faults. But with dramatically slower NV-DIMMs sharing a memory channel with DRAM, I think I'd rather prefer storage semantics in most cases, because DIMM access is synchronous and thus blocking.
Once you have a dedicated software layer, the actually interconnect via memory channels or PCIe becomes a bit less of an issue and the future is supposed to be CXL anyway. Which brings me to the question: how will that deal with excessive latencies on memory-mapped storage?
P.S.: how far ahead is there even support for Optane in Intel memory controllers? -
msroadkill612 ur above my pay grade, but yes, i was going to say "virtual memory", & yes - if anything that sounds simpler to adapt software to than mixing nand into ram slots.Reply
To this newb it seems surreal that intel cunningly made their strategic optane weapon so exclusive, that nobody came onboard, & a potential hit product just withered away. -
abufrejoval
Surreal or not, a potential monopoly holds such terrible attraction an IT company will spend billions to get there.msroadkill612 said:To this newb it seems surreal that intel cunningly made their strategic optane weapon so exclusive, that nobody came onboard, & a potential hit product just withered away.
Intel's earliest trauma was losing the DRAM market to Japanese commodity makers in the mid 1980's.
And then Intel has learned from IBM's mainframe business which maintained ECL-class exclusivity even after switching to CMOS commodity technology.
It also owes most of its success to being able to kick out the proprietary Unix market with the price point that commodity PC based ISA technology offered.
Intel consistently dreams up ways of selling exclusive value at commodity production cost. They always dream big and they have failed big quite a few times already. And because the industry knows exactly what Intel is trying to achieve, often enough they have refused to go where Intel wanted them (e.g. Itanium, Xeon Phi, Omnipath), because they knew they'd have to pay heavily once the dependency was there. -
msroadkill612
Ta for the cool history lesson.abufrejoval said:Surreal or not, a potential monopoly holds such terrible attraction an IT company will spend billions to get there.
Intel's earliest trauma was losing the DRAM market to Japanese commodity makers in the mid 1980's.
And then Intel has learned from IBM's mainframe business which maintained ECL-class exclusivity even after switching to CMOS commodity technology.
It also owes most of its success to being able to kick out the proprietary Unix market with the price point that commodity PC based ISA technology offered.
Intel consistently dreams up ways of selling exclusive value at commodity production cost. They always dream big and they have failed big quite a few times already. And because the industry knows exactly what Intel is trying to achieve, often enough they have refused to go where Intel wanted them (e.g. Itanium, Xeon Phi, Omnipath), because they knew they'd have to pay heavily once the dependency was there.
We may well be witnessing an historic mortal miscalculation this time, with their haughty view of any threat by AMD to the core of their hegemony.
If it is as u say, the big buyers must love the new deal & embrace it warmly.