Another Record Broken: 6 Gb SAS, 16 SSDs, 3.4 GB/s!

New LSI Controllers Plus 16 X25-E Flash SSDs Equal 3.4 GB/s!

In late July, we published the article Breaking Records With SSDs: 16 Intel X25-Es Do 2.2 GB/s, which we created after being encouraged by this YouTube video. It is a fun bit of documentation on the initial project that was tackled by Paul Curry, who, like us, went through the efforts for the purpose of promoting decent SSDs. The goal was to illustrate unprecedented storage throughput.

Although we were lucky enough to deliver even better performance numbers on our SSD RAID array (16 Intel X25-E SSDs versus 24 Samsung PB22-J drives), we weren’t really satisfied, and decided to do some more testing using other HBAs and RAID controllers. Intel and LSI came to the rescue, supplying the latest MegaRAID 9210-8i (Intel RS2BL080) and 9260-8i cards. Man the battle stations!

More Bandwidth, Please

The 2.2 GB/s result we achieved already sounds pretty impressive, but some simple math reveals that this number could even be higher, since each of the 16 Intel X25-E flash SSDs is realistically able to provide more than 220 MB/s of throughput. The theoretical maximum for our array should be somewhere around 3.5 GB/s, which is 60% more than what we’ve reached so far. Clearly, we were looking at some sort of bottleneck.

Platform? Check.

Our platform, a Supermicro X8SAX X58 motherboard and an Intel Core i7-920 2.66 GHz quad-core processor with 3 GB DDR3-1333 memory, is definitely fast enough to support higher bandwidth numbers. After all, we used two x16 PCI Express 2.0 slots for the controllers. Each of the 16 lanes is able to serve up 250 MB/s bi-directionally, providing up to 4 GB/s each way on the first generation of PCI Express. On the X58 platform with PCIe 2.0, this is doubled to 8 GB/s. Clearly, the platform wasn’t the issue.

Controllers? Replaced!

We initially used Adaptec’s 5808 cards, which are high-performance RAID HBAs offering balanced performance and a plethora of software-oriented features, used to manage the RAID array. One of our very first checks was the potential bandwidth of these cards, since they employ eight PCI Express 1.1 lanes to interface with the system. But two cards running eight times 250 MB/s still results in 2 GB/s each, or 4 GB/s total. Between two cards, we should have had enough available bandwidth. But it turns out that we didn't.

We decided to use LSI’s latest HBA and RAID products instead. LSI sent us its latest MegaRAID 9260-8i cards, which are SAS 6 Gb/s boards. In addition, Intel became intrigued by the possible performance gains and provided two other cards, namely LSI’s 9210-8i, which it also sells under the Intel brand, named RS2BL080. These aren’t yet available, and they don’t come with a powerful XOR engine, cache, or kick-butt enterprise features. But both cards are among the first HBAs to utilize PCI Express 2.0, which effectively doubles the interface bandwidth using eight PCIe lanes. With this hardware, we were pretty darned confident that we’d be able to break our earlier performance numbers—and we did!