E-ATX Dual LGA2011-V3 Mobo with 5(+) x16 Speed/Slot widths,Gen3 PCIE slots?

Toggle sidebar Toggle sidebar
C

chriscambridge

Commendable
Aug 5, 2016
33
0
1,540
Hi,

We are trying to locate an E-ATX motherboard which can host dual V3 Xeons, and has 5 (+) PCIE slots that are x16 size, and run at x16 speed at Gen3.

We keep finding boards that are using multiplexers/PCIE switches so that they do not actually offer the full x16 speed at Gen3.

Does anyone know of any motherboards that can run 5 GPUs at full speed at x16 width on Gen3?

It is our understanding that each Xeon will offer 40 PCIE lanes, and the chipset used will offer more additional lanes, so we guess the most at full speed that can be on the mobo would be five PCIE slots.

Please note, the space in-between the slots does not matter as we use x16 to x16 powered PCIE risers within a GPU rack.

Thanks in advance for any help or advice.
 
Sort by date Sort by votes
Lutfij

Lutfij

Titan
Moderator
Oct 7, 2009
54,303
4,818
173,390
Which V3 Xeons are we talking about here? Regarding the lanes, if you take dual GPU's into consideration from a single processor, you are left with 8 lanes when each PCIe x16 slot is allocated 16 lanes(16x2=32). The rest 8 lanes are fed to the last/third slot. In essence you can work with 5 slots each running at 8 lanes when working with a 40 lanes processor. You can read more here.

You could look at Supermicro's solutions since they have been known to take custom orders for boards however if that feature is missing by the CPU manufacturer and the allocation of such lanes by manufacturer then the motherboard manufactures have nothing to do. Further reading here.

You may be interested in this board as well or this.
 
Upvote 0 Downvote
C

chriscambridge

Commendable
Aug 5, 2016
33
0
1,540
Thanks for the information.

We are/ will be, using dual Xeon E5-2683 V3 processors, which gives 40+40+ the PCIE lanes given by the chipset.

We calculated that 5 GPUs would equal, 5 *16 = 80 PCIE lanes, leaving a few left over.

All the mobos we have seen by Supermicro are proprietary form factor sizes and not E-ATX.

We have an over sized ATX GPU rack, hence why an E-ATX would fit best would us having to make any/many modifications.

The Z10PE-D16 WS must use PCIE switching, which halves true speed, as clearly with a single Xeon with 40 PCIE lanes you would not be able to run four GPUs at 16 speed. (64 PCIE lanes required); Not unless the chipset offered 24 lanes which seems quite a lot.

The D-8 only offers two GPUs slots at x16 speed.

Much better than these two, if you do not mind x8 speeds and/or PCIE switching, is the

X99-E-10G WS

https://www.asus.com/Motherboards/X99-E-10G-WS/specifications/

Which can run 5 or 7 GPUs at the following speeds; however PCIE switching is in use once again, as the necessary PCIE lanes are not available on a single processor.

(1) x16
(2) x16,x16
(3) x16,x16,x16
(4) x16,x16.x16,x16
(7) x16,x8,x8,x8,x8,x8,x8

** Surely there must be a motherboard that actually only offers PCIE slots for available PCIE lanes, and fully utilises all the lanes from the CPU and Chipsets on offer? Excluding Supermicro due to there non standard form factor sizes.
 
Upvote 0 Downvote
C

chriscambridge

Commendable
Aug 5, 2016
33
0
1,540
We are running a 1080, and three 970s.

"even a 1080 doesnt benefit from 16 lanes over 8"

That probably add's up if you use GPUs for gaming; we don't, we use them for data processing, where data transfer is important.

This is why we want full speed/slot width gen3 on all 5 slots.
 
Upvote 0 Downvote
C

chriscambridge

Commendable
Aug 5, 2016
33
0
1,540
So far the motherboard that seems to fit this requirement is the Supermicro X10DRG-Q (Intel C612) Workstation Motherboard (LGA2011-V3)

[4 PCI-E 3.0 x16 (double-width) slots + 2 PCI-E 3.0 x8 (1 in x16) slots]

This review on this mobo from ServeTheHome states that the full 80+ PCIE lanes are available to the slots

"Four dual slot PCIe x16 slots that do not have components underneath to obscure. Solid cooling layout. 80 PCIe 3.0 and 4 PCIe 2.0 lanes exposed to users via slots which is best in class"..

https://www.servethehome.com/supermicro-x10drg-q-review-gpu-compute-server-motherboard/ (review)

https://www.supermicro.com/products/motherboard/Xeon/C600/X10DRG-Q.cfm (product info)

https://www.supermicro.com/manuals/motherboard/C612/MNL-1677.pdf (pdf manual)

The only thing is that it is large, and therefore we would have to modify the GPU rack to be able to take a board this big.
 
Upvote 0 Downvote
Lutfij

Lutfij

Titan
Moderator
Oct 7, 2009
54,303
4,818
173,390
The links you've posted are broken however nothing a quick edit or re-post can't fix ;)

I think you selected my post as the Best Answer prematurely since the threads has some more progression left. As opposed to what you said, Supermicro can actually make the board of your choice and their support team are on hand to give you some sound advice. I'm also certain they also provide racks to accompany their boards.
 
Upvote 0 Downvote
C

chriscambridge

Commendable
Aug 5, 2016
33
0
1,540
Hi Lutfij,

We picked yours as the solution as all the mobos we have looked at so far, the only ones that seem to have true, full PCIE slots to lanes are the Supermicros.

But I hear what you are saying about possible other solutions, so have deselected this for now. If nothing better is offered then I will pick yours as the best answer, to be fair, and because I cannot select my own answer! :)

Thanks for the update on the broken links - all fixed now.

In terms of chassis/rack, we want to keep our existing GPU rack.
 
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom