Facebook asked for denser and cheaper 100TB QLC NAND SSDs during its keynote address at the Flash Memory Summit. Coincidentally, an hour later, Toshiba laid out its ambitious plans to develop 100TB QLC SSDs. Toshiba plans to pioneer the use of QLC (Quad-Level Cell) technology with a new variant of its 64-layer BiCS3 NAND, but the company will also employ its TSV (Through-Silicon Via) technology to boost its die stacking capability, which effectively doubles capacity per package. QLC NAND makes several tradeoffs, particularly in the endurance and performance department, but it will provide more density than even the beefiest new SSDs.
Of course, the real question for we enthusiasts is if we can expect to see massive QLC-powered SSDs in our PCs. Let's take a closer look at the tech, then work out the math to see if QLC SSDs are possible for normal users (hint: they are).
The First Step: Stack NAND Die Higher
NAND vendors stack up multiple die inside of the familiar NAND packages we are accustomed to seeing inside SSDs. Manufacturers connect NAND die with wire bonding, but the tiny wires are delicate, and thus stacking a large number of die is a tricky proposition that can result in ruined NAND packages.
The pictures above illustrate a typical wire-based die stack from two angles. Most vendors only stack up to eight die to minimize the chance of damage, but they use 16 die stacks on rare occasions for premium products. However, stacking 16 die also results in performance loss due to the increased impedance of longer wires. The reduced signal integrity forces the vendors to slow down the flash's clock speed. Samsung recently added a small F-chip to its 48-layer NAND to combat the problem, but that adds another component to the device, and thus complexity and cost.
Toshiba developed TSV technology for NAND to circumvent the problem (similar to the technique fabs use for HMB). TSVs connect the die through vertical electrical connections in the center of the die. The channels mate with microbumps on an underlying interposer that is perched atop a substrate. This technique eliminates the performance-killing side effects.
It also purportedly makes stacking the die less risky and allows the vendors to stack them in a perfect vertical column, as opposed to the offset technique used with wire bonding. TSV reduces impedance, and Toshiba explained that it also boosts sequential read/write throughput by 1.7x/3.1x, respectively, in power-restricted environments. Unfortunately, TSV tends to be a more expensive manufacturing technique than wire bonding.
TSV reduces NAND's core and I/O power consumption 45% during high-speed data transfers, which equates to a 30/40% improvement during moderate read/write activity (at the device level). Reducing power consumption has the knock-on effect of reducing the amount of heat the NAND generates, which is good for thermal-constricted environments (such as with M.2 SSDs). Fabs also employ thermal microbumps between the die to enhance heat dissipation. We've actually observed TSV NAND in working demos at FMS 2015 in the PMC-Sierra booth (a third-party SSD vendor).
Perhaps most importantly, TSV allows the company to increase the height of the stack up to an unheard-of 32 die, which increases density. Toshiba's 64-layer BiCS3 NAND packs in 32 GB (256 Gbit) per die, so a stack of 32 dies will provide 512 GB in a single NAND package. The company also announced that its next-generation BiCS4 NAND (coming in 2017) will have a 64 GB (512 Gbit) density. Combining TSV with BiCS4 will provide up to 1 TB in a single package.
The Second Step: Add More Bits
SLC NAND stores one bit per cell, which is represented by two voltage states, and MLC doubles that to two bits per cell with four voltage states. TLC NAND, which has become the most prevalent type of 3D NAND, stores three bits per cell and has eight voltage states.
QLC NAND will store four bits per cell with 16 voltage states--a feat that would be impossible with planar NAND. The lithography of planar NAND has shrunk to the point that each cell holds roughly a dozen electrons, and the cells wear out quickly. Luckily, 3D NAND has a relaxed lithography (3x nm), so the cells are strong enough to handle the increased stress associated with programming and erasing 16 voltage states. Even with the benefits of 3D NAND, the increased stress results in less endurance. Toshiba provided a slide that outlines the increased density of the voltage states.
Voltages tend to drift over time, which spawns read errors. Sophisticated LDPC algorithms help improve the raw bit error rate of TLC NAND, which increases endurance, but QLC NAND will require powerful ASICs to handle the increased error correction overhead. Interestingly, Toshiba has its own proprietary QSBC (Quadruple Swing-By Code) error-correction algorithms that are purportedly more powerful than standard LDPC.
Toshiba did not specify a P/E (Program/Erase) ratio for its proposed QLC NAND, but if we compare its proposed SSD to the data Facebook presented during its keynote (more on that later), we can safely assume it is in the range of 150 cycles. This leads to an exponentially lower endurance rating compared to normal SSDs. Most NAND vendors do not disclose the P/E cycle rating of their NAND, but planar MLC/TLC is widely thought to be able to absorb roughly 3,000/1,000 cycles, respectively, but it varies with the type of error correction employed.
What A QLC SSD Looks Like
Toshiba indicated that it had a QLC NAND-based SSD in development and presented a slide with specifications.
With up to 100 TB of flash at its disposal, it makes sense that the QLC SSD employs a PCIe 3.0 interface compared to the SATA interface found on an HDD. The broader data lanes allow the QLC SSD to deliver up to 3/1 GBps of sequential read/write throughput and 50,000/10,000 random read/write IOPS. For a PCIe SSD, the random write specifications are hardly earth shattering. QLC NAND will be used only to store infrequently-accessed WORM (Write Once Read Many) data, such as archival and cold data, so random write performance is a minor concern.
Reducing Power Consumption
A 100 TB SSD would reduce the amount of space required to store large amounts of data, which is a task that commonly falls to the lowly HDD. Even with hefty 8 TB HDDs, it requires 12 drives to provide the same amount of storage as one QLC SSD. The QLC SSD sips power at 9W and, most importantly, has a 100mW idle power consumption with a 5ms resume time (wake). A single HDD consumes the same amount of power as the QLC SSD during operation, but its performance pales in comparison. The drive also requires 8W during idle to provide a similar resumption time. HDDs have lower idle power states, but they trade resumption time for reduced power consumption.
To make matters even more lopsided, most data centers deploy HDDs in RAID arrays, which multiplies the idle power consumption of one drive by the number of drives in the array (96W for HDD compared to 0.1W with the QLC SSD). The low idle power is important for WORM data, as cold data storage devices spend most of their time in an idle state. HDDs simply cannot compete. Writing or retrieving data from an array usually requires all of the drives to spin up, which increases active power consumption dramatically.
Endurance Concerns
On the surface, the 3-6 PB of endurance for the QLC SSD appears impressive, but we compiled comparative data from other enterprise SSDs to highlight the drastically reduced endurance. SSD endurance relies upon a complex mixture of factors, such as the type of flash used, the amount of overprovisioning on the SSD, and its error correction capability. We do not have any of these specific measurements for the Toshiba QLC SSD, but we can use the basic data provided by Toshiba to generate ballpark figures.
| Header Cell - Column 0 | Toshiba QLC SSD | Samsung PM963 | Intel DC P3500 | Intel DC P3600 | Intel DC P3700 | Micron 9100 | Micron 9100 MAX |
|---|---|---|---|---|---|---|---|
| NAND | Toshiba QLC | 48-layer 3D TLC NAND | 20nm MLC | 20nm MLC | 20nm MLC | 16nm MLC | 16nm MLC |
| Maximum Capacity | 100TB | 3.84 TB | 2 TB | 2 TB | 2 TB | 3.2 TB | 2.4 |
| Endurance | 3000-6000 | 5466 | 1095 | 10950 | 62050 | 3280 | 6570 |
| TB's of Endurance Per TB Of Addressable Capacity (Lifetime) | 30 - 60 | 1,423 | 547 | 5,475 | 31,025 | 1,025 | 2,738 |
| DWPD | 0.027 - 0.054 DWPD (est. 3 year) | 1.3 (3 years) | 0.3 (5 years) | 3 DWPD (5 years) | 17 DWPD (5 years) | 0.56 (3 years) | 2.5 (3 years) |
We included Intel's DC SSD family because it spans from low-endurance to high-endurance products. The QLC SSD will offer 0.027 to 0.054 DWPD (Drive Writes Per Day) of endurance if we assume a three-year warranty period (which isn't confirmed), whereas high-endurance SSDs provide up to 17 DWPD. Even "normal" low-endurance SSDs weigh in with 0.3 DWPD.
Another way to view endurance data is to calculate the TBs of data that a user can write per TB of user-addressable capacity, as noted in the chart. It is clear that the QLC SSD lags the field by a large margin. QLC's endurance-per-TB is so low that it will not be suitable for small-capacity SSDs in enterprise applications.
Can We Use QLC SSDs In The Desktop? Sure, Why Not.
Consumer SSDs come in all shapes and sizes, and they also come with varying amounts of endurance. The industry is trending to low-endurance 3D TLC NAND SSDs because they provide more than enough performance and endurance for most users, and come with a nice price point to match.
| Header Cell - Column 0 | Toshiba QLC SSD | Samsung 2TB 850 EVO |
|---|---|---|
| NAND | Toshiba QLC | 48-layer 3D TLC NAND |
| Capacity | 2 TB (est) | 2 TB |
| Endurance | 60 - 120 | 150 |
The Samsung 850 EVO has sold more units than any other SSD in history, so it makes sense to use the 2 TB version as our baseline comparison model. There are faster and more endurant SSDs on the market, but the 850 EVO hits the perfect mix for the majority of applications.
The 2 TB 850 EVO features a 150 TBW rating, which means it can absorb 150 TB of writes over its five-year lifespan. A theoretical 2 TB QLC SSD isn't far behind with a rough maximum of 120 TBW (generated from our endurance-per-TB of addressable capacity calculations). This amount of endurance is enough for most desktop users, and building bulkier QLC SSDs (such as 4 and 8 TB models) will boost the endurance even more.
For instance, a 6 TB QLC SSD would offer up to 360 TBW of endurance. The key to QLC's success will be high capacities--smaller versions won't pass endurance muster. The minimum 60 TBW is a bit less encouraging, but SSD manufacturers can add more overprovisioning to boost endurance. We haven't run into any issues with the 850 EVO's relatively low endurance threshold, and QLC SSDs might be close enough to be useful.
The 850 EVO also has a big performance advantage over the theoretical QLC SSD, but extra overprovisioning can solve the bulk of this problem. The 850 EVO's stellar performance is a rarity in the value segment (hence its popularity), but other SSD vendors are already transitioning to low-cost DRAM-less SSDs and "HDD Replacements," which won't be hard for QLC SSDs to match on the performance front.
Facebook Calls For WORM SSDs...Again
It's likely that Toshiba already has one enthusiastic customer lined up. Three years ago at the Flash Memory Summit, Facebook famously asked the industry to develop cheap flash for WORM applications. It may have taken three years, but it appears that Toshiba granted the company's wish.
Facebook and Toshiba did not acknowledge that they are working together, but Facebook coincidentally mentioned that it is already experimenting with 100 TB QLC SSDs (it even had a graphic of the SSD). It also called out a target of 150 P/E cycles. Toshiba reportedly has QLC SSDs in early field deployments, and it's easy to assume that Facebook is one of the leading customers.
Facebook split its infrastructure into multiple tiers of storage based upon the speed of each respective solution, and true to its reputation of being the industry's most prolific early technology adopter, it is already adopting 3D XPoint for the fastest tiers. Facebook is accustomed to demanding that the industry develops new technology; sometimes it even forces the issue, as evidenced by its Open Compute Project (OCP). For now, the company asked for more M.2 SSDs, broader NVMe adoption, next-gen Non-Volatile Memories (such as 3D XPoint) and more QLC. The speaker also noted that he would like to see five-bit-per-cell NAND after the industry climbs the QLC mountain.
The QLC Ecosystem Expands
Toshiba is working feverishly to develop QLC NAND, which means that its foundry partner WD/SanDisk will also eventually have the same base material (though it will have its own unique QLC-based products). WD already has several system-level products, namely the Active Archive and InfiniFlash platforms, that would benefit from QLC SSDs. Micron mentioned during its Flash Memory Summit keynote that it is also working on QLC NAND, but due to the somewhat fractured IMFT alliance, it is unclear if Intel will have access to the technology.
We interviewed Samsung executives at the show, but the company emphatically insisted that 3D TLC NAND is the answer, and it is not currently productizing QLC NAND. With its overwhelming lead in the SSD market, it is hard to question Samsung's decision.
SK hynix, the only remaining fab, has been silent on the QLC front.
Several NAND vendors indicated that the increased capital expense and R&D associated with moving beyond 64-layer 3D NAND would likely lead to longer developmental cycles with each node. Much like Intel's new tick-tock-tock scheme, the companies will use the extra time to refine the architecture, which may be the key QLC-enabling ingredient. The third-party SSD ecosystem is also paving the way for a QLC-powered future; several controller vendors reported that they have QLC-compatible ASICs in development for both enterprise and consumer applications.
The Days Of QLC Are Almost Upon Us
I remember the resistance when the industry shifted from SLC to MLC NAND. At the time, a friend of mine (a respected SSD engineer) vehemently insisted that MLC was unsuitable for data center use (which I found amusing). There was a similar backlash as we moved to TLC NAND, and there will likely be even more resistance during the transition to QLC NAND.
However, the cost benefits are too large to ignore, and QLC NAND is inevitable. New 3D NAND fabs are finally coming online from all the major vendors, which will flood the market with good, cheap NAND. Adding QLC NAND to the mix might just boost capacity enough to begin to dislodge nearline HDDs out of archival roles, which will change the storage game entirely.
More importantly for us, QLC SSDs can, and likely will, make an appearance in the desktop. SSD vendors usually build scale, and thus realize the economies of scale that drive down cost, in the consumer market; a QLC SSD for the masses may be a key part of Toshiba's strategy. The SSDs will likely only come in very large capacities, but they will certainly come with a hefty discount that may put the final nail in the HDD coffin.
