Sign in with
Sign up | Sign in
Another Record Broken: 6 Gb SAS, 16 SSDs, 3.4 GB/s!
By ,
1. New LSI Controllers Plus 16 X25-E Flash SSDs Equal 3.4 GB/s!

In late July, we published the article Breaking Records With SSDs: 16 Intel X25-Es Do 2.2 GB/s, which we created after being encouraged by this YouTube video. It is a fun bit of documentation on the initial project that was tackled by Paul Curry, who, like us, went through the efforts for the purpose of promoting decent SSDs. The goal was to illustrate unprecedented storage throughput.

Although we were lucky enough to deliver even better performance numbers on our SSD RAID array (16 Intel X25-E SSDs versus 24 Samsung PB22-J drives), we weren’t really satisfied, and decided to do some more testing using other HBAs and RAID controllers. Intel and LSI came to the rescue, supplying the latest MegaRAID 9210-8i (Intel RS2BL080) and 9260-8i cards. Man the battle stations!

More Bandwidth, Please

The 2.2 GB/s result we achieved already sounds pretty impressive, but some simple math reveals that this number could even be higher, since each of the 16 Intel X25-E flash SSDs is realistically able to provide more than 220 MB/s of throughput. The theoretical maximum for our array should be somewhere around 3.5 GB/s, which is 60% more than what we’ve reached so far. Clearly, we were looking at some sort of bottleneck.

Platform? Check.

Our platform, a Supermicro X8SAX X58 motherboard and an Intel Core i7-920 2.66 GHz quad-core processor with 3 GB DDR3-1333 memory, is definitely fast enough to support higher bandwidth numbers. After all, we used two x16 PCI Express 2.0 slots for the controllers. Each of the 16 lanes is able to serve up 250 MB/s bi-directionally, providing up to 4 GB/s each way on the first generation of PCI Express. On the X58 platform with PCIe 2.0, this is doubled to 8 GB/s. Clearly, the platform wasn’t the issue.

Controllers? Replaced!

We initially used Adaptec’s 5808 cards, which are high-performance RAID HBAs offering balanced performance and a plethora of software-oriented features, used to manage the RAID array. One of our very first checks was the potential bandwidth of these cards, since they employ eight PCI Express 1.1 lanes to interface with the system. But two cards running eight times 250 MB/s still results in 2 GB/s each, or 4 GB/s total. Between two cards, we should have had enough available bandwidth. But it turns out that we didn't.

We replaced Adaptec’s popular 5-series RAID 5805 cards with two LSI controllers.We replaced Adaptec’s popular 5-series RAID 5805 cards with two LSI controllers.

We decided to use LSI’s latest HBA and RAID products instead. LSI sent us its latest MegaRAID 9260-8i cards, which are SAS 6 Gb/s boards. In addition, Intel became intrigued by the possible performance gains and provided two other cards, namely LSI’s 9210-8i, which it also sells under the Intel brand, named RS2BL080. These aren’t yet available, and they don’t come with a powerful XOR engine, cache, or kick-butt enterprise features. But both cards are among the first HBAs to utilize PCI Express 2.0, which effectively doubles the interface bandwidth using eight PCIe lanes. With this hardware, we were pretty darned confident that we’d be able to break our earlier performance numbers—and we did!

2. The SSDs: Sixteen Intel X25-Es

Intel is in the process of ramping up its next generation of flash SSD drives, as all new products are based on 34 nm MLC flash (X25-M for consumers). 

The X25-E drives should also be receiving an upgrade with the die-shrunk flash memory chips soon. Nevertheless, we kept the same drive lineup that we had used for the initial story: 16 X25-E enterprise-class flash SSDs from Intel with a capacity of 64 GB each. The total capacity of a fully-configured array with these 16 drives is exactly 1 TB, but capacity wasn’t our objective here. Rather, we wanted more throughput and better I/O performance.

The Intel X25 SSDs are all based on an in-house design with a 10-channel flash controller and integrated 16 MB cache memory. They have Native Command Queuing (NCQ) support, allowing the drives to optimize performance, manage wear leveling, and counter performance-degrading effects such as write amplification. In short, the SSDs continually optimize storage to maximize the user experience at all times.

None of the SSDs currently available support Serial Attached SCSI (SAS) or 600 MB/s transfer speeds, and it appears that a faster interface isn’t necessary--at least not yet. The drives we used are sufficient to reach 3GB/s+ throughput in RAID, but we believe they cannot actually saturate the new generation HBAs and RAID controllers utilizing PCI Express 2.0 and SAS/600.

3. Controllers: LSI 9260-8i (RAID Controller) And 9210-8i (HBA)

LSI MegaRAID 9260-8i

Let’s look at the naming of these new controllers first. The designation “8i” stands for eight internal ports; “9200” is the new PCI Express 2.0 RAID-on-Chip generation; and RAID 6 is supported, which you can tell from the second part of the model number (“60”). 

LSI offers two internal SFF-8087 MiniSAS connectors, which host four SAS/SATA connections each. The card is a low-profile design, which can also be deployed into compact 2U rackmount servers. You’ll just have to bend the cables, as the connectors point upwards towards the top of the rackmount enclosure.

Connectivity

LSI totally redesigned its architecture to meet the demands of SAS/600 environments. The RAID-on-Chip (RoC) architecture is built around an LSI SAS2108, which includes the x8 PCI Express 2.0 interface and SAS/600 connectivity (6 Gb/s).

It can access a 512 MB DDR2-800 cache memory, and a battery backup module (BBU) was already installed on our samples (though LSI says that the BBU is still an optional component). The BBU prevents data loss in case of power failure while unwritten data is still in the controller’s cache memory.

Performance?

According to LSI, the controller itself supports a maximum read throughput of 2,875 MB/s, and a peak write performance of 1,800 MB/s. Knowing that we’re using two controllers and Windows software-based RAID to create an array across two controllers (and two different PCI Express links) we’re clearly on the safe side regarding bandwidth.

LSI supports RAID levels 0, 1, 5, and 6 with double redundancy, as well as nested RAID levels 10, 50, and 60. We recommend looking at the controller’s datasheet on the LSI Web site for more details.

LSI MegaRAID 9210-8i

The 9210-8i, aka Intel RS2BL080, offers the same number of SAS/SATA ports as the 9260-8i, and the PCB has exactly the same dimensions, but the product is somewhat different, and much simpler overall. It does not feature a powerful RAID-on-Chip solution with XOR acceleration for RAID 5 and RAID 6; it does not come with cache memory; and it doesn’t feature a battery backup unit. 

However, that strip-down may be helpful in terms of throughput, as there is no processing unit and no cache memory that might be trying to improve performance while effectively limiting it in our scenario (as was the case with Adaptec’s RAID 5805). Of course, it also helps with pricing, we're sure.

We don’t have much information on the 9210-8i, as it has not yet been officially released. It also utilizes PCI Express 2.0 (eight lanes) as one of the first products on the market, and it is ready for SAS/600 drives as well.

Creating a RAID 0 Stripeset on LSI’s 9200-8i Cards

The MegaRAID Storage Manager lists all LSI controllers and attached devices.

Once a RAID array has been created, it will also list the volumes, called Virtual Drives.

Creating an array can be done using simple commands, or by choosing the advanced option.

Once all drives are added into a new drive group, you can set parameters and policies. These were the settings we used.

4. RAID Creation, Test Setup, And Throughput

RAID Creation

Since the two RAID controllers had different capabilities and features, we decided to take different approaches for each.

In the case of the fully-featured MegaRAID 9260-8i, as well as the Adaptec RAID 5805 we used before, we created RAID 0 arrays with eight SSDs on each of the controllers. Then we used Windows Vista’s RAID capabilities to build a software-based RAID 0 array using the two RAID 0 volumes. This way we’d be utilizing eight SSDs on each controller and on each PCI Express 2.0 link.

For the MegaRAID 9210-8i, we found that the easiest (and best-performing) solution was one huge RAID 0 array managed by Windows Vista. So, we went into the Storage Manager and created a software RAID 0 array using all 16 available Intel X25-E flash SSDs.

Test Setup

System Hardware
Hardware
Details
CPU
Intel Core i7-920 (45 nm, 2.66 GHz, 8 MB L2 Cache)
Motherboard (Socket 1366)
Supermicro X8SAX
Revision: 1.0
Chipset Intel X58 + ICH10R
BIOS: 1.0B
RAM
2GB DDR3-1333 Corsair CM3X1024-1333C9DHX
System HDD
Seagate NL35 400 GB
ST3400832NS
7,200 RPM, SATA/150, 8 MB
Controller I
2 x Adaptec RAID 5805
8 Port SAS Controller 512 MB Cache
Controller II
2 x LSI MegaRAID 9210-8i (aka Intel RS2BL080)
8 Port SAS HBA
Controller III
2 x LSI MegaRAID 9260-8i
8 Port SAS Controller 512 MB Cache
Power Supply
OCZ EliteXstream 800W
OCZ800EXS-EU
Benchmarks
I/O Performance
IOMeter 2006.07.27
Fileserver-Benchmark
Webserver-Benchmark
Database-Benchmark
Workstation-Benchmarks
Streaming reads
Streaming Writes.
System Software & Drivers
Driver
Details
Operating System
Windows Vista Ultimate SP1
Intel Chipset
9.1.0.1007
AMD Graphics
Radeon 8.12
Intel Storage Drivers
Matrix Storage Drivers 8.7.0.1007


You will find very detailed information on our storage reference test system in the article Tom’s Storage Charts 2009: A New Test Environment.

Throughput

These are really great results. Both LSI controllers take our RAID arrays, using the 16 Intel X25-E flash SSDs, to a whole new level of throughput as they jump over the 3 GB/s level! The 3,005 MB/s average throughput is excellent for a RAID controller, which typically is better at I/O than at high throughput. LSI’s new 6 Gb/s HBA, the 9210-8i, managed to reach more than 14% better throughput, hitting 3,432 MB/s average read throughput across all drives.

The write throughput wasn’t really different than the read results on the LSI MegaRAID 9210-8i, but the 9260-8i RAID card incurred a slight impact. Still, that’s nothing to worry about—this array on those two controllers is probably faster than anything most of you (or us) have ever seen in the storage arena.

5. Benchmark Results: I/O Performance

The new LSI 9260-8i might not have been optimized for a database type workload yet, as it fails to deliver the performance level of Adaptec’s RAID 5805. We found the simpler 9210-8i from LSI more impressive in this benchmark, as it beats the Adaptec card, delivering almost 45,000 I/O operations per second. The database benchmark consists of 67% read and 33% write access using only 8 KB blocks.

Once we switch to the file server scenario, the cached and fully-featured MegaRAID 9260-8i does better, outperforming both its little brother and the Adaptec RAID 5805. The file server benchmark pattern is based on many different block sizes between 512 bytes and 64 KB, with 80% read and 100% random operation. You need an intelligent engine to optimize this for type of access.

What a result! The Web server test does not involve any write operations, which is favorable for performance. LSI’s uncached 9210-8i blows away the two others, but keep in mind that the RAID 0 is rather insecure. All of these benchmarks should be taken as experimental, providing an outlook on the possible performance levels, rather than representing a truly useful scenario.

Both LSI cards are much faster in our workstation benchmark pattern, delivering 54-73% better performance than the same SSDs on the Adaptec RAID 5805.

Please read the article Tom’s Storage Charts 2009: A New Test Environment for detailed information on the benchmarks.

6. Conclusion

Simply replacing the Adaptec RAID 5805 cards with the brand new product generation from LSI makes it clear that the RAID controller was a real bottleneck for our 16 SSD armada. Granted, the Adaptec cards are still feature-rich and balanced enterprise-class SAS HBAs, but the LSI MegaRAID 9260-8i and 9210-8i (aka Intel RS2BL080) are the first storage cards to utilize PCI Express 2.0 rather than 1.1, which doubles the bandwidth of the x8 slot from 2 GB/s to 4 GB/s. In addition, both cards now also support SAS/600 and SATA/600, but this has no impact on our performance results at this time, as all flash SSDs are still based on SATA/300.

The SAS HBA MegaRAID 9210-8i is not yet commercially available, but should be released rather soon.

These cards allowed us to reach a maximum read throughput of 3.5 GB/s and an average of 3.4 GB/s on our test system by simply adding all 16 SSDs into one huge RAID 0 array. All I/O performance benchmarks depend a bit on CPU performance, which is why the fully-featured LSI 9260-8i may be quicker in some I/O benchmarks (not in all, though). We tested on the latter card using RAID 0 hardware arrays and a RAID 0 stripe set that we based on the two RAID 0 volumes. This product was limited to 3.0 GB/s throughput.

In the end, it was easy to exceed the former throughput record of 2.3 GB/s, but it wouldn’t have worked without LSI’s support in providing two 9260-8i cards and Intel forwarding the brand new 9210-8i cards. Tweaks such as processor overclocking could certainly further improve performance, but having more bandwidth on the PCI Express side, as well as on the individual storage ports, makes clear that the SSD vendors now have to come up with faster solutions. The architecture is here. Now we're waiting on faster drives.