Multiple RAID Controllers, mdadm on RHEL DAS Server?

[killerhertz]

I have a storage server (RHEL 5.2) and (2) 2U DAS chassis containing 24 drives each (48 drives total). The server has dual LSI 9280-8e SAS RAID controllers. My goal for this setup was to aggregate performance across both RAID controllers, but I'm having trouble doing so.

To troubleshoot I first connected each controller to (1) 2U DAS so each was able to see 24 drives. I setup a RAID0x24 for each. After benchmarks using Iozone, I was seeing sustain writes of 1650 MB/s for each controller. LSI advertises 1800 MB/s writes with their LSISAS2108 chip, so I'm pretty much right at the margin.

In order to combine the throughput across the controllers (1650 Mb/s * 2 = 3300 MB/s max) I created a RAID0 array in mdadm. When I do this, I'm still seeing a max rate of 1650 MB/s, which doesn't make sense. I'm wondering if I'm missing something. I've tried various strip sizes for the hardware and software RAID with no success.

 

[tokencode]

You may be hitting a different bottleneck. The cards you are using are PCIe 8x cards, which will have a max throughput of approximately 2GB, which is not far off from 1650MBs if there is some additional overhead. If you're motherboard only has 16 available PCIe lanes and you have another PCIe device in use that could also be consuming part of your bandwidth. I would confirm that your motherboard supports greater than 2000MB/sec on the PCIe bus (or your target of 3300MB/sec).

 

[killerhertz]

You may be hitting a different bottleneck. The cards you are using are PCIe 8x cards, which will have a max throughput of approximately 2GB, which is not far off from 1650MBs if there is some additional overhead. If you're motherboard only has 16 available PCIe lanes and you have another PCIe device in use that could also be consuming part of your bandwidth. I would confirm that your motherboard supports greater than 2000MB/sec on the PCIe bus (or your target of 3300MB/sec).

It's a dual IOH-36D motherboard, so 36*2=72 pci-e lanes total. Both slots are x8. But you're right, I believe there is some other bottleneck that is holding it back, unless the motherboard is total junk...