Expansion card lets you insert 512GB of extra DDR5 memory into your PCIe slot — CXL 2.0 AIC designed for TRX50 and W790 workstation motherboards

Wouldn't this be a lot slower than regular memory on the board? Isn't pcie 5 slower?

Phillip0web said:

Wouldn't this be a lot slower than regular memory on the board?

For sure, it would. According to this, CXL latency is expected to be 170-250 ns.
https://www.servethehome.com/compute-express-link-cxl-latency-how-much-is-added-at-hc34/
I think CXL 3.0 cuts down on that. But, it won't work any miracles.

Probably the main way that something like this would be used is page-migration. Pages can be swapped into local DRAM, either on first access or as they start to heat up. Cold pages can be demoted, sort of like virtual memory.

Here's a resource I found, concerning the topic. I haven't really looked at it, so I have no idea whether you'd find it helpful. It does at least touch on all the keywords and core concepts.
https://jiyuan.is/papers/asplos25-m5.pdf

Phillip0web said:

Isn't pcie 5 slower?

As for the bandwidth question, PCIe 5.0 x16 is on the order of how fast a desktop dual-channel memory setup runs. Yes, a bit slower, if you're only accessing in mostly one direction. Of course, server (currently) have up to 12-channel memory subsystems. However, the nice thing about CXL is that it scales. So, if you have a server platform, with 64 CXL lanes, you can add 4 of these cards and get 4x linear scaling.

What's really cool is that later revisions of CXL support switching. So, it lets you aggregate large amounts of memory in a pool that might be larger than the total amount of DRAM you could connect directly to the CPUs.

What's even better is that CXL defines cache-coherent access between host CPUs and accelerators, meaning a GPU could pull data directly from these cards, without having to go to the host CPU. Cache-coherent means that the CPU can update data on the card, without having to worry about whether the GPU will see the latest version (or vice versa).

bit_user said:

For sure, it would. According to this, CXL latency is expected to be 170-250 ns.

Reminds me of how putting DRAM on plug-in cards used to be the 'normal' thing.

RAM always took lots of space, even when it was very low capacity, a 64KB magnetic core memory card was a full sized Unibus card in a PDP-11: those invented the 19" racks.

S-100 µ-computer systems also put RAM into a different slots by default, the CPU card really just had the CPU with it's required support circuitry.

Apple ]










Probably the main way that something like this would be used is page-migration. Pages can be swapped into local DRAM, either on first access or as they start to heat up. Cold pages can be demoted, sort of like virtual memory.

Here's a resource I found, concerning the topic. I haven't really looked at it, so I have no idea whether you'd find it helpful. It does at least touch on all the keywords and core concepts.
https://jiyuan.is/papers/asplos25-m5.pdfCXL has a very long-term vision and with that the shortcomings you get when you try to design very far ahead.

One of the main drivers was that in a typical HPC environment, you may find yourself having plenty of left-over RAM on one machine, that runs with high locality and a small memory footprint, while another really needs to access a huge dataset with poor locality, but can' afford swapping to disk or even NV-RAM without slowing down by orders of magnitude.

So CXL allows to scale-out PCIe compatible, but with in a multi-master (root) environment, that changes allocations at run-time. The idea is to lend/loan resources like RAM to/from other machines in the same data center, especially when they are close by, or use one of the optical interconnects CXL was envisioned for, to overcome the copper energy and latency issues, that mount up as distances grow. It's also designed to fit with UCI, which handles PCIe semantics/abstracions within silicon dies, so scaling can be done up and down.

Of course, it's not just RAM, storage, GPUs, any type of PCIe device could get elevated via CXL to out-of-the-box sharing and connectivity, but that causes layers, abstractions, indirections all of which cost raw speed and power.

Which is at least one reason, why proprietary links like NVlink, or what the Chinese are cooking, aim either for better performance or a smaller patent pool to pay licenses for.

And yes, either your application is directly conscious of the resources and their topology and migrates resources like storage (including RAM) to wherever they are most adequate for the workload, or they require proxies/agents to do it for them: traditionally that would be an OS responsibility, in scale-out it's a new layer, because you can't teach 1970 Unixoids with a God complex new tricks.

bit_user said:

As for the bandwidth question, PCIe 5.0 x16 is on the order of how fast a desktop dual-channel memory setup runs. Yes, a bit slower, if you're only accessing in mostly one direction. Of course, server (currently) have up to 12-channel memory subsystems. However, the nice thing about CXL is that it scales. So, if you have a server platform, with 64 CXL lanes, you can add 4 of these cards and get 4x linear scaling.

What's really cool is that later revisions of CXL support switching. So, it lets you aggregate large amounts of memory in a pool that might be larger than the total amount of DRAM you could connect directly to the CPUs.

What's even better is that CXL defines cache-coherent access between host CPUs and accelerators, meaning a GPU could pull data directly from these cards, without having to go to the host CPU. Cache-coherent means that the CPU can update data on the card, without having to worry about whether the GPU will see the latest version (or vice versa).

Yes, all that rich functionality with all the proper abstractions... and the overhead that implies. Nvidia will claim it's all futile and wasted effort and their proprietary fabric the only path worth following; the hyperscalers cook their secret sauces or even do their own optical switching interconnects (Google).

For you as a consumer, I'd say these CXL RAM cards will only become interesting, once they wind up used and really cheap on eBay.

bit_user said:

Many desktop motherboards have featured PCIe 5.0 x16 since 2021 (when Intel launched LGA 1700 with it), but those don't support CXL.

But not x16/x16, no non-Threadripper or EPYC board supports it, and even then not all CPUs do, and even then not all of them support CXL, and even on boards that do not all PCIe slots support CXL, like the Gigabyte TRX50 AI TOP.

Even if desktop boards did expose support for CXL you'd be limiting performance of anything stronger than a 5090 by using x8 (Techpowerup's PCIe scaling test shows the 5090 is -just- at the limit of x8 5.0), and hampering performance of the AIC.

Maybe we need to go back in time ..

1. Motherboard was just lots of expansion slots.
2. Cartridge style cpu sockets that conform to an industry standard connection to support
Intel, AMD, ARM, NVidia, Chinese chips
3. Couple of dedicated memory slots for memory riser cards.
4 .Couple of dedicated NVMe riser cards slots.
5. Various 16x 8x 4x slots to plugin whatever controller you need, audio, wi-fi, lan, SATA,SAS, in my opinion 1x slots obsolete

If you don't need audio, wi-fi, 10 USBs why have them eating up PCIe lanes etc.
Imagine a dual socket E-ATX with all those lovely slots to fill ?

@das_stig my old p8z68-v has more support than my new h670 board... but she have 3 nvme... just cables to use these nvme as usefull will cost two boards. The old system are more responsive don't have tons of cpu power or IO but is enough to put the 13600T to sleep!