12 Gb/ SAS: Welcoming The Next Generation Of Enterprise Storage
Early in 2009, we were introduced to the first 6 Gb/s SAS products. Four years later, we're ready for a look at 12 Gb/s-capable SAS devices from LSI and HGST. For a true appreciation of this technology, we need to take a deeper look into how it was conceived.
When it comes to evaluating an interface, it's really to overlook all of the details. From the highest levels, most folks assume a theoretical doubling of speed. But speed is relative. Are clock rates increasing? How about the number of available channels? Do latencies change? And what about line code? As we've learned from the evolution of PCI Express, it's not easy to simply jump from 2.5 to 5 to 10 GT/s. At some point in there, physics becomes an increasingly large obstacle.
In fact, PCIe shares some of the same characteristics as SAS and SATA. From the first to second generations of PCI Express, data rate increased from 2.5 to 5 GT/s. Both standards employed 8b/10b encoding with simple, fixed transmitter equalization. Third-gen PCIe is limited to 8 GT/s. However, overhead is significantly reduced through a transition to 128b/130b. At the end of the day, peak theoretical throughput doubled, though the path to get there was different.
We could have guessed, then, that the creation of a 12 Gb/s SAS standard wasn't going to be easy for the T10 technical committee.
SATA-IO board member Paul Wassenberg gave us some great insight when he answered why the SATA protocol would not be going to 12 Gb/s:
"Twelve gigabit per second SATA would seem to be the logical next step, and the T10 (SAS) committee has done a lot of the work on 12 Gb/s already. From this work, we know that the transition from 6 Gb/s to 12 Gb/s is not simple. SAS 3.0 (12 Gb/s) requires transmitter equalization, which adds a great deal of complexity to the interface controller and the PHY. In silicon, complexity equates to more die area, which means higher cost. Also, the protocol needs to change to support transmitter training, and that turns out to be fairly significant. Additionally, many of the backplanes and cables that worked fine at 6 Gb/s won't reliably carry data at 12 Gb/s."
Low-cost client systems don't readily embrace technologies that cost more. But the enterprise does, particularly when performance is of the utmost importance. For many customers, the transition to 12 Gb/s SAS will be evolutionary, and they'll integrate it one piece at a time over the next few years. With this in mind, we're missing the one puzzle piece that probably means the most in the SAS ecosystem: 12 Gb/s expanders.
Particularly when you're talking about mechanical disks, it takes a lot of drives to saturate an eight-port 6 Gb/s HBA. But IT professionals carefully balance the controllers, expanders (which adapt a certain number of SAS ports to a larger number of storage devices), and drives themselves to optimize for their specific application. If you have an eight-port HBA or RAID card today, it's limited to 48 Gb/s (8 x 6 Gb/s). Hooking up 128 disks via an expander severely limits the throughput of each one.
On the other hand, adopting 12 Gb/s SAS increases that ceiling to 96 Gb/s (at least in theory). By replacing HBAs and expanders, it's possible to alleviate the interface-imposed bottleneck without even changing out your drives.
Of course, removing one bottleneck creates another one somewhere else. So, where would the weak link be in a 12 Gb/s-capable SAS topology? It turns out to be the PCI Express bus.
That's right. An eight-port 12 Gb/s HBA or RAID card saturates an eight-lane PCI Express 3.0 link. Why not use a 16-lane connector? Most 2U servers still ship with up to x8 upgrade slots, so 16-lane cards aren't on the table right now.
Naturally, testing 12 Gb/s SAS requires that we use compatible adapters and drives. LSI supplied us with its SAS 9300-8e controller, and HGST provided the already-public Ultrastar SSD800MM.
