The Hardware
Intel Optane DIMM
3D XPoint will come to market as SSDs in M.2, U.2, and AIC form factors, along with memory DIMMs. The die could ride in BGA or TSOP packages, though we have only seen BGA. We also expect Intel will use 3D XPoint for on-package Xeon and KNL (Knights Landing) implementations, which should make their debut with the second-generation Purley platform. It remains to be seen if 3D XPoint is suitable for on-die use.
Intel brought along its Optane DIMM to Storage Visions 2016, which was our first sighting of an actual 3D XPoint-powered product. The device is electrically and physically JEDEC DDR4 compliant so that it can plug into a standard DIMM slot. However, there are a range of methods to deploy DIMMs, such as for storage or memory. Using 3D XPoint efficiently as memory will require a modified DDR4 interface, which we will cover later.
A single Optane DIMM can reach up to 512GB, giving a dual-core server the potential for up to 6TB of RAM-like memory. The NVDIMM has an onboard FPGA under the heatsink to manage the underlying 3D XPoint media. Intel would likely transform the FPGA into an ASIC as development moves along. However, it is noteworthy that we have not seen any performance demos of 3D XPoint NVDIMMs used as memory, and the last we've seen or heard of them was in January 2016.
The disappearing act doesn't do much to quell the rampant rumors that indicate the leading-edge material will unexpectedly not have enough endurance for heavy-use memory applications (DIMMs). There are numerous techniques to mitigate wear, such as effective hot spot management on the device or system-level caching and tiering techniques. Intel is in a unique position to lay the groundwork for these techniques with its control of the IMC.
3D XPoint DIMMs will also require encryption, because physical attacks are much easier with a persistent memory. An attacker could just remove the DIMMs and retrieve the data. This same technique is possible with regular DRAM sticks, but isn't feasible and certainly not comparable to the ease of recording the data from an Optane DIMM. Intel has not indicated what method it would use to enable on-the-fly encryption without hampering performance.
Intel Optane SSD
Optane SSDs have been the star of every Intel demo, but unsurprisingly, the company kept them hidden from our prying eyes until very recently at IDF 2016. We had originally spotted the Optane in July 2016 on the UNH-IOL Integrators list, which is an industry group that verifies NVMe compatibility, so we knew it was in the offing. Intel had its speedsters on the IDF show floor, but both booths had the cards facing downwards, which shielded us from the view of the top of the card. The cards were in a workstation and a QCT (Quanta Cloud Technology) sled, which is used in data center environments. We attempted, with frustration, to take pictures from multiple angles, but finally found the cards in an upward-facing position after one of the final presentations.
We pine so eagerly for shots of the top because Intel has a history of releasing client SSDs based on enterprise SSDs, so this same design will almost certainly be available for us enthusiasts, albeit with a skull tattooed upon it. The prototype card connected to the host via PCIe 3.0 x4 but only featured 140GB of capacity, which we expect to increase as the products come to market. Intel and Micron are working with early samples, so we expect capacity to scale quickly as development progresses.
A glance at the edge of the PCB reveals only four visible 3D XPoint packages, but there are empty pads for many more. The pads do not appear to conform to standard ONFI-compliant BGA designs. After we examined the empty pads on the QuantX prototype (below), we are confident the companies are using the same BGA alignment, and likely the same custom media interface.
Intel has already progressed from a custom FPGA (a developmental platform) to an ASIC. Intel's proprietary ASIC will be a big advantage as 3D XPoint matures, as Intel doesn't have to deal with an outside company, which can hinder development and time to market. All told, Optane will consist of custom drivers, firmware, controllers, and "optimized interfaces," which we assume refers to the many layers of proprietary interconnects that span from the 3D XPoint packages to the host interface. Interestingly, Optane's ASIC isn't even in the same ballpark as Micron's QuantX, at least in one workload, which we will get to in the performance demos further on.
Micron QuantX SSD
Micron, on the other hand, is leveraging an exclusive relationship with a "third party SSD controller vendor" to produce the engine for its 3D XPoint SSDs. We were able to maneuver our way into some exclusive pics, which revealed the CNEX branding that let the cat out of the bag.
Micron's developmental prototype features two daughterboards with two BGA-mounted 3D XPoint packages apiece. The BGA mounts for the empty 3D XPoint pads apparently do not adhere to standard ONFI 4.0 ball spacing templates, and we know it uses a proprietary DDR-like interface.
Micron's board featured 128GB of total capacity. The knowledge that it is a single-deck material with 8GB per die either suggests more 3D XPoint packages mounted to the bottom of the daughterboards, or that there are four die stacks per package.
The final devices will likely be single-PCB affairs, and Micron noted they would weigh in at a top capacity of 1.6TB. This seems like a low capacity point compared to today's 6TB NAND-based SSDs, but the low-QD performance scaling helps to defray the need for the excess capacity, which users often purchase because heftier capacities provide more random workload performance.
QuantX features a wider PCIe 3.0 x8 connection in comparison to the Optane x4 connection, but as we will see in the performance demos, even an x8 interface holds the media back from reaching full performance. The prototype also includes some features we didn't expect on a 3D XPoint SSD, such as five Micron DRAM packages and capacitors for power loss protection. DRAM is still faster than 3D XPoint, so Micron can employ it to manage the underlying L2P (logical to physical) mapping or as a persistent NVRAM cache, which increases speed. Any DRAM on an enterprise-class product requires a power back up, hence the capacitors.
Micron guarantees at least 25DWPD of endurance based upon a five-year warranty. We also note the embedded QSFP+ connector on the prototype board for NVMeoF (NVMe Over Fabrics), which uses the native NVMe protocol over a network connection to provide the best possible latency from remote storage devices. The networking angle ties nicely into the CNEX story.
The CNEX Connection
Micron covered lettering on its prototype PCB with a yellow sticker, but the daughterboard tipped us off to CNEX labs due to the emblazoned "CNEX - Diamond - v2.0" branding. Neither company is willing to comment directly, but it's pretty hard to dispute that CNEX is involved, as controllers are the company's specialty and its name is literally written all over the prototype.
CNEX may be a little-known name, but the company is on an upward trajectory with investors such as Micron, Seagate (of course), Samsung, and Cisco. Being on the ground floor of 3D XPoint development is a big plus, but CNEX has numerous tricks up its sleeve, including a pending series of controllers that enable native Ethernet connectivity (which may be the purpose of the QFSP+ connector on the QuantX prototype). The controllers also support bleeding-edge LightNVM and Open Channel initiatives that change the SSD paradigm, or they can serve as a plug-and-play NVMe platform. CNEX also happens to have a proprietary lightweight, deterministic ECC solution that seems to fit Micron's descriptions perfectly.
These topics are a bit much to chew over here, but head over to the CNEX Labs Looks Like Missing 3D XPoint Puzzle Piece article for a deep dive on the CNEX design. CNEX recently scored a patent for remote access of memory over Ethernet networks with NVMe controllers, so the company is certainly building a trove of protected IP that provides Micron with numerous options for QuantX deployment. CNEX may just be the next big third-party SSD controller vendor in the making.
MORE: All Enterprise Storage Content
MORE: All SSD Content
