Truth About PCI Express Bandwidth Usage- Nvidia Tesla / Quadro Setup

blokeandsally

Distinguished
Nov 24, 2010
15
0
18,510
As the title states, I am trying to determine the real deal when it comes to the amount of bandwidth needed to run my system at top performance. I am currently building a ws system with 2x Nvidia Teslas, 1x Quadro GPU, and would like to run my LSI x8 SAS/SATA 6gps RAID card as well. The problem I seem to keep returning to is what I need in terms of PCI Express support. Everything I am running is PCI Express 2.0. I have noticed that unless you look at an Intel 2011, or Dual Xeon/Opteron setup, there are really no other options that support a full speed setup. So I guess the question I have is "what do I need in a mobo"?

I have looked at the Nvidia Maximus setups, and most seem to follow the x16,x8,x8 platform. I have read many times that these cards are not capable of full bus saturation, and that even the Tesla server setups seem to run x16,x8,x8. When I work the RAID card into the mix I then need an additional slot for the RAID card which is rated for x8. If i use a board that is x8,x8,x8,x4 will that be sufficient bandwidth for the setup? If so should I put the RAID card in one of the x8 slots, and one of the GPU or compute units in the x4?

The boards I am currently looking at are the ASUS Sabertooth, ASUS Maximus, and the ASUS Crosshair. I am also looking at a dual G34 board from ASUS. the other question is what CPU would be the most cost effective in terms of performance/cost ratio. I have no problem with AMD or Intel. I also have a spare i5 quad core SB that I could make use of, but that is only an option if I can get away with lower PCI Express bandwidth. I don't want to cheap out too much on the board, after laying out the type of cash i did on the GPU and Compute units. However I don't need to blow the budget.

Does anyone have any advice re what would be the appropriate setup for the cards? i would also appreciate any recommendations regarding motherboards as well.
 

zyky

Distinguished
Sep 12, 2006
174
0
18,710
How much the limited PCI-e lanes limits your performance depends mostly on your workload.
If you're sending a lot of small tasks using a large dataset to the compute cards, you're going to suffer more than code that has longer threads. Additionally, if you're using mechanical hard drives rather than SSDs, you're likely not going to benifit much from having the raid controller on a x8 link.

And yes, if you need PCI-E lanes, you should be looking at LGA 2011 or dual/quad 1366 systems.
 

blokeandsally

Distinguished
Nov 24, 2010
15
0
18,510
thanks for your quick response. I will be using Samsung 840 Pro SSD drives, and the workload will be largely smaller chunks of data over longer periods of time. Do you have any recommendations for a 2011 motherboard? Would an Xeon E5 or i7 be any better suited for the job?
 

zyky

Distinguished
Sep 12, 2006
174
0
18,710
If the cards are sent small chunks of data, left to process it for a long time, and then return a result, then you should be fine w/o needing lots of PCI-E bandwidth/lanes. If you can interlace the workload so that two cards are computing while one is returning/fetching new data - you can use a multiplexer and still have roughly equivilent performance to having actual pci-e lanes available.

This is exactly why SLI/crossfire can perform well even when using a PLX chips to multiplex more pci-e lanes onto less.

You could try looking for a board featuring a NF200 or PLX8747.