Micron Unveils 32GB NVDIMMS At Supercomputing 2017

Micron announced at Supercomputing 2017 that it increased the density of its DDR4 NVDIMMs (Non-Volatile Dual Inline Memory Module) to 32GB. The increased density comes on the heels of Micron's 8GB and 16GB DIMMs. 

NVDIMMs, also known as persistent memory, provide, well, persistence for data stored in DRAM. That means the module can permanently store the data in DRAM even after an unexpected power loss. Micron accomplishes this feat with an RDIMM equipped with both DRAM and NAND. In the event of a power loss, the NVDIMM flushes the data from the DRAM to the onboard NAND storage.

Persistent memory enables explosive DRAM acceleration for workloads with stringent data protection requirements, such as in-memory and traditional databases, metadata storage, HPC, or caching for storage solutions, among others. You can access the DIMM as either storage (block mode) or memory (DAX - direct access mode), with the latter offering the best performance.

SSDs can also be used to accelerate some workloads, but they only operate as a storage device. They also have much lower performance, and intense workloads can radically reduce the lifespan of the underlying flash. As expected, DRAM unlocks much higher levels of performance (~20X) and also provides nearly unlimited endurance. Also, when used as a block device, NVDIMMs can boost SSD endurance by absorbing write workloads before flushing write-combined data to NAND.

Like most DIMMS, you insert the JEDEC-compliant NVDIMM into a memory slot. You can use several NVDIMMs in a system but special population rules apply, and notably, the system requires at least one standard DDR4 "chaperone" DIMM. Micron says that most of its customers require 64GB of NVDIMM storage, so reaching the 32GB capacity point allows its customers to use two DIMMs instead of four, freeing up more room for standard DDR4 memory.

NVDIMMs come in -N, -F, and -P flavors. All of these models feature the same core principle of combining DRAM with NAND onto the same module, but they present different capabilities to the host system. Micron's NVDIMM-N allows the system to access the DRAM as either block-based storage, much like a normal storage device, or as normal memory with load/store semantics (DAX - Direct Access). The onboard 64GB NAND package is not mapped to the host.

While NVDIMMs can be used as block-based storage, DAX offers far superior performance because it removes the I/O stack bottleneck. Micron generated the performance data above with its 8GB NVDIMM-N model. It offers 76.4x more performance than an NVMe SSD when used in DAX mode. NVDIMMs are also immune to the performance degradation we see with SSDs after extended workloads, so they easily offer 10x the performance of some NAND-based SSDs.

In the event of a power loss, the system sends a save/restore command to the DIMM via the Save-n pin. This command initiates a data flush from the DRAM to the NAND. The Save_n pin is the safest and most reliable method of issuing the command, but there are numerous other techniques, such as through APIs and sideband signals. The 64GB NAND package operates in pSLC (Pseudo SLC) mode, so it only offers 32GB of storage capacity, but pSLC increases write performance and reduces flushing duration. A tethered PowerGEM supercapacitor (pictured on the 8GB model) provides enough power to flush the data to NAND.

For the 64GB model, Micron offers the same PowerGEM form factor as the previous-generation models, but it requires more time to flush the much larger DRAM capacity to NAND. For users with stricter requirements, Micron now offers a larger capacitor that can maintain a 30-second flush window. Micron has a new generation of capacitors coming to market that have higher power density, thus offering a faster flush window with the smaller form factor, but the company hasn't announced the release date.

In the past, most NVDIMMs were installed into existing servers. The bulky supercapacitors require dedicated cabling to each DIMM, which hampered adoption in the earlier stages of development. Micron also provides the option to route a dedicated 12V UPS power supply to the module, which is a differentiator compared to some other NVDIMM products. In-chassis UPS batteries (such as those used in some OCP designs) are becoming popular, and the ability to leverage the existing hardware instead of adding another component is always desirable. The dawn of the Skylake server era found most OEMs offering off-the-shelf NVDIMM-capable systems. That was one of the key ingredients to fostering further ecosystem support, which we'll dive into shortly. 

For all of their promise, NVDIMMs have taken time to mature. The original products topped out at 2133 MTps, thus limiting the performance of the other DIMMs in the system. The original model featured a discrete MUX in that data path which limited speeds to 2400 MTps, whereas the new model incorporates the MUX into the DRAM itself and can provide up to 2933 MTps. The integrated MUX clears the signal path and allows for higher data transfer rates, which Micron says is an advantage borne of its fabrication capabilities.

The 32GB NVDIMMs consume ~7-8W, while a similar DDR4 module consumes ~4.5-7W. Most of the additional power consumption comes at the hands of the integrated FPGA controller and the NAND package.

NVDIMMs operate with a JEDEC-standardized interface and protocol, but operating system support was an early challenge. Unlocking the potential of persistent memory requires operating system awareness, and to that end, Microsoft developed new cache and memory management instructions such as CLFLUSHOPT (Optimized Cache Line Flush) and CLWB (Cache Line Write-Back) to supplement CLFLUSH (Cache-Line Flush). The instructions manage and ensure data flushes. The pmem.io set of NVM libraries also allows developers to optimize applications based on the DAX features in both Windows 10 and Linux, which has also helped more applications become persistent memory aware. 

Persistent memory is different than storage-class memory, such as 3D XPoint, but it is already available in DIMM form factors. NVDIMMs also offer the best overall performance and come with the economics of proven technologies. The wide availability of NVDIMM-capable OEM systems is also pushing the technology out from the Tier 1 hyperscale data centers, which were the main early adopters, to the mainstream. Of course, broad application support for persistent memories will take years, but now the tools are readily available.

Micron says it is continuing to enable the ecosystem and that the higher capacity models should help increase the uptake. Micron's 32GB models are available now.