Dual Xeon Motherboard (Socket 2011) recommendation

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510
Hi all!
Always wanted to build a dual xeon workstation and now it seems my build will finally be realised. A while ago I bought two E5-2670s 8-core xeons together for less than 200 euros. Because I already own all other parts only 1 thing is remaining, selecting the right mobo!

Trying to decide...
Asus Z9PE-D8
Asus Z9PE-D16
ASRock EP2C602

What do you guys recommend? Any experience with one of the above mobo's? In the future I would like to make a decent hackintosh build with these parts.

Parts...
Cpu: 2x E5-2670
Cpu cooler: Corsair H100i, Corsair H80i
Ram: 32GB 1600 Non-ECC Corsair Vengeance
Ram: 64GB 1333 ECC Hynix (future upgrade?)
Graphics: 1x 760 Reference, 1x 980ti Evga Hybrid
Pcie SSD: OCZ Revodrive 350
Soundcard: Asus Phoebus
Network: Intel Pro 1000PT Dualport
Psu: Enermax MaxRevo 1350W
Case: Corsair 900D
 
Solution
"Doesn't the Z9PE-D8 have x16 bandwidth on all slots? And is 1600Mhz ECC RAM for dual cpu's necessary or just highly recommended? I would prefer to buy RAM later on and for now use the non-ecc corsair vengeance ram. I'm a huge fan of re-using / switching around parts and because I already own a Modded Revodrive 350 (bought it for 200 euro) I do not want to switch it out even for a performance gain. Plan to skip the Intel Pro 1000PT if no driver issue occurs with the 82574L (need it for another build).

And well I was thinking about the layout and this is what I came up with:
slot 1 - SC Asus Phoebus
slot 2 - SSD OCZ Revodrive 350
slot 3 - Empty (add usb card or a third gpu)
slot 4 - Empty (add usb card or a third gpu)
slot 5 - GPU...

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510


There is also a cheaper D16-2L 2 port version also with an i350 Nic, would be better since I only need 2 ports. I will install Windows 10 Pro and OSX on it so using link aggregation just like my 2012R2 server won't be a option/needed. The D8 has much more IO like usb ports and an onboard ALC898 (crappy audio but suppprted for osx). Altough I can always add usb cards. What do you think about the ASRock board? Also less IO and I like the price point (not many pcie ports tough).
 

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510


Yes I agree, then I will try to get my hands on a Z9PE-D8. Could be difficult, most shops sold out or just 1 remaining. I also watched this video from teksyndicate about the board (https://www.youtube.com/watch?v=7Tyudcr1uLY). They were talking about this board being able to use the max turbo speed as base clock speed. And because the 2670s can turbo up to 3.3Ghz this can be very interresting :).
 
Spartan70NL,

The ASUS Z9PE-D8 is a good choice from the viewpoint of producing good CPU benchmark scores. On Passmark baselines, the best CPU result is using a Lenovo D30 and the two Xeon E5-2670's have a score of 20855. Second place is the ASUS Z9PA-D8 EUR 287 @ amazon.de) at 20817, 3rd ASRock EP2C602 (EUR 325) (20795), 4th Supermicro X9DRL-3F/IF (20697), a server motherboard without PCIe x16, and 5th Supermicro X9DAi (EUR 568)(20694).

Motherboard Layout:
Have a good look though at the various choices to ensure that you can use the PCIe slots as required. If you are having two double-height GPU's, a PCIe SSD, PCIe soundcard, and Network card, the arrangement of the slots is essential to be correct because the double height GPUs will cover the adjacent slot. the ASRock is not good as the first x16 slot is too close to RAM slots. On the ASUS Z9PA, the first slot next to the RAM slots is x8 but the card can not be longer than the slot as it will not clear another set of RAM slots. GPU 1 and GPU 2 will both cover the other two x8 slots, so there is not a slot to fit the OCZ Revodrive or the soundcard. The Z9PA is ATX and really, an E-ATX format is necessary. Looking at that situation, the Supermicro X9DAi, which is the larger E-ATX format has the best board layout, with 3X PCIe x16 with three x8 slots between and proper clearance in from of the first x8- which could be used by the Revodrive.

RAM: If you are using two CPU's, the RAM will need to be PC3-12800R, that is DDR3-1600 ECC registered. The registered RAM has a parity check that acts a kind of buffer to, in effect, synchronize the two processors.

The Xeon E5-2670 /2680/2690 series was excellent and are fantastic bargains today, thanks to a large number that were cycled out of servers by firms such as facebook and google.

I use, for analysis, simulation, and rendering, a dual Xeon E5-2690 system that performs very well and was very economical. This was based on buying a used HP z620 for $270 US, the two processors ($4,100 new) for $306, 64GB of RAM for $128, and so on. I re-used a Quadro K2200 from another system with a Tesla M2090 6GB GPU coprocessor ($86) and the 3D visualization performs similarly to a Quadro M5000 8GB- a $1,900 GPU. The total cost was about $1,300 and at the moment has the highest system for an HP z620.

What kind of work are you doing?

Cheers,

BambiBoom


CAD / 3D Modeling / Graphic Design:

HP z420 (2015) (Rev 3) > Xeon E5-1660 v2 (6-core @ 3.7 / 4.0GHz) / 32GB DDR3 -1866 ECC RAM / Quadro K4200 (4GB) / Samsung SM951 M.2 256GB AHCI + Intel 730 480GB (9SSDSC2BP480G4R5) + Western Digital Black WD1003FZEX 1TB> M-Audio 192 sound card > 600W PSU> > Windows 7 Professional 64-bit > Logitech z2300 2.1 speakers > 2X Dell Ultrasharp U2715H (2560 X 1440)
[ Passmark Rating = 5581 > CPU= 14046 / 2D= 838 / 3D= 4694 / Mem= 2777 / Disk= 11559] [6.12.16]

Analysis / Simulation / Rendering:

HP z620 (2012) (Rev 3) 2X Xeon E5-2690 (8-core @ 2.9 / 3.8GHz) / 64GB DDR3-1600 ECC reg) / Quadro K2200 (4GB) + Tesla M2090 (6GB) / HP Z Turbo Drive (256GB) + Samsung 850 Evo 250GB + Seagate Constellation ES.3 (1TB) / Creative Sound Blaster X-Fi Titanium PCIe sound card / 800W / Windows 7 Professional 64-bit > Logitech z313 2.1 speakers > HP 2711x (27" 1980 X 1080)
[ Passmark System Rating= 5675 / CPU= 22625 / 2D= 815 / 3D = 3580 / Mem = 2522 / Disk = 12640 ] 9.25.16
[ Cinebench R15: OpenGL= 119.23 fps / CPU = 2209 cb / Single core 130 cb / MP Ratio 16.84x] 10.31.16
 

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510


Hi Bambiboom, thanks for your comprehensive response really appreciate it. Nice workstations you have there :p, what kind of mobo's do you have? Those HP workstations are great prebuild systems especially on the inside, have seen one of those at my work also with 2670s. I will try to purchase a Z9PE-D8, there are several confirmed hackintosh builds on tonymacx86. Also looked at the X9DAi it's a nice mobo however it had not enough pcie slots and no confirmed hackintosh builds.

This system will mostly be used for graphic design (more datamerge than design though), changing and trying out parts and some gaming. The main reason behind this build is mostly fun, but I would like to work more at home and might look into 3d modeling in the future. My old/current hackintosh build is socket 1155 based (I7-3770K/Asus ROG V Extreme), which does not have enough pcie ports and not enough performance to my liking, hoping this build will be overkill.
 


Spartan 70NL,

You're very welcome.

The HP motherboards are made by HP, and I assume they are designed in-house, but there is an idea in the air that they also consult Supermicro on some level. However they are designed, the performance results are very good and the integration of the system makes these very reliable and extremely quiet. There are a number of Lenovo D30's, HP z820's and Dell Precision T7600 systems with higher CPU marks than a number of ASUS ZPE-D8's.

Do have a good look at the Z9PE-D8 layout to ensure that all the peripherals will fit. I'm quite convinced that, if there are two double-height GPU's, the only PCIe slot available will be an x8 which is only usable if the card is not longer than the slot. If the intention is to add a PCIe Revodrive, soundcard and network card, I think the solution would be to use instead of the Revodrive an M.2 on an adapter card, skip the additional network card- use the onboard connections, and have an external USB sound interface. Anyway, a Samsung 960 Evo 1TB has better performance than the $1,900 US Revodrive and for $480- about 1/4th the cost.

Interesting project!

Cheers,

BambiBoom
 

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510



Doesn't the Z9PE-D8 have x16 bandwidth on all slots? And is 1600Mhz ECC RAM for dual cpu's necessary or just highly recommended? I would prefer to buy RAM later on and for now use the non-ecc corsair vengeance ram. I'm a huge fan of re-using / switching around parts and because I already own a Modded Revodrive 350 (bought it for 200 euro) I do not want to switch it out even for a performance gain. Plan to skip the Intel Pro 1000PT if no driver issue occurs with the 82574L (need it for another build).

And well I was thinking about the layout and this is what I came up with:
slot 1 - SC Asus Phoebus
slot 2 - SSD OCZ Revodrive 350
slot 3 - Empty (add usb card or a third gpu)
slot 4 - Empty (add usb card or a third gpu)
slot 5 - GPU 980ti Hybrid (connect)
slot 6 - GPU 980ti Hybrid (block)
slot 7 - Risercable GPU 760 (mount side-ways in the case)

Although I could switch this around if the cards don't fit, all of them have (custom) backplates so I have to keep that in mind.
 
"Doesn't the Z9PE-D8 have x16 bandwidth on all slots? And is 1600Mhz ECC RAM for dual cpu's necessary or just highly recommended? I would prefer to buy RAM later on and for now use the non-ecc corsair vengeance ram. I'm a huge fan of re-using / switching around parts and because I already own a Modded Revodrive 350 (bought it for 200 euro) I do not want to switch it out even for a performance gain. Plan to skip the Intel Pro 1000PT if no driver issue occurs with the 82574L (need it for another build).

And well I was thinking about the layout and this is what I came up with:
slot 1 - SC Asus Phoebus
slot 2 - SSD OCZ Revodrive 350
slot 3 - Empty (add usb card or a third gpu)
slot 4 - Empty (add usb card or a third gpu)
slot 5 - GPU 980ti Hybrid (connect)
slot 6 - GPU 980ti Hybrid (block)
slot 7 - Risercable GPU 760 (mount side-ways in the case)

Although I could switch this around if the cards don't fit, all of them have (custom) backplates so I have to keep that in mind."


Spartan70NL,

PCIe Slot Layout: While the Z9PE-D8 WS has seven PCIe x16 slots, and the two Xeon E5's provide 80 PCIe lanes, 7 X 16 = 112, so not all slots will run at x16. The specification of the lanes' disposition is:

4 x PCIe 3.0/2.0 x16 (dual x16 or quad x8)
2 x PCIe 3.0/2.0 x16
1 x PCIe 3.0/2.0 x16 (x8 mode)

Looking at the motherboard diagram: I think you could make:

Slot

1 = SC Asus Phoebus
2 = SSD OCZ Revodrive 350
3 = GPU1
4 is covered by GPU1
5 = GPU2
6 is covered by GPU2
7 Could have a riser/cable for a third GPU, but personally, I'd suggest having two higher performance GPU's - perhaps sell all the cards and have two GTX 1070's and besides higher performance with Pascal GPU's. 16GB. and 3,840 CUDA cores, that way both will run at 3.0 x16, the installation is straightforward, and there is less heat.

Cheers,

BambiBoom


 
Solution

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510


Hi BambiBoom, thanks for explaining I had not taken into account the max amount lanes available for the Xeons :S. Think I would prefer your layout, and maybe upgrade gpu's when necessary later on. Really appreciate your help, you sure have alot of experience with workstation builds. I'm still not sure about the ram, is PC3-12800R ECC or ECC in general (1333 or 1600) actually needed for a dual xeon system? Or won't it work (properly) at all without ecc?
 


Spartan70NL,

Sorry for my late reply.

For the Xeon E5-2670, the native RAM memory speeds are DDR3 800/1066/1333/1600 and ECC is supported..

PC3-12800R ECC = DDR3-1600 ECC registered.

The ASUS Z9PE-D8 however, supports:

8 x DIMM, Max. 64GB, DDR3 2133(O.C.)/2000(O.C.)/1866*/1600/1333/1066 MHz ECC, Non-ECC, Un-buffered Memory
Quad Channel Memory Architecture
*For Registered Memory, Max. 256GB DDR3 1866/1600/1333/1066/800 MHz
** Refer to www.asus.com for the Memory QVL (Qualified Vendors Lists).

Notice that the limited for ECC, Non-ECC, Un-buffered Memory is 64GB, but using ECC registered, the maximum is 256GB.

Because the ASUS Z9PE-D8 can use both the Xeon E5-2600 first series and the faster v2 series, for which the native RAM speed is 1866, and in respect of the 4X maximum total RAM using registered, and because all dual Xeon systems I've ever heard of use registered, consider buying for the proposed system, 4X 16GB PC3-14900R (= 64GB DDR3-1866 ECC Registered).

That means that in the future, the E5-2670's may be changed to a pair of, for example, Xeon E5-2667 v2, which are 8-core @ 3.3 /4.0GHz- and that use DDR3-1866. These are $900 US these days (2.17) but in two years, perhaps $300, just as the Xeon E5-2670 in 2012 cost $1,550 and in 2016 at one point was selling for only $65.

I am planning to sometime sell the two systems I use and buy a Supermicro Superworkstation SYS-7037A-i , which cost about $750 in the US. These provide the case, motherboard, CPU coolers, 900W power supply, have 16 RAM Slots for up to 1TB DDR3-1866, and support Xeon E5-2600 first version and v2, 3X double height GPU's. this might be a good option for your system. With these you just plug in the CPU's. RAM, GPUs, and drives and it's ready to go. And in that I would use a pair of Xeon E5-2687w v2's- 8-core @ 3.4 /4.0GHz, a Quadro P4000 8GB -these should be our soon, 128GB of RAM, a fast M.2 500GB + a 1TB SATA SSD storage drive. I can't find a price for the SYS-7037A-i in NL or DE, but you might have a look becase the Asus Z9PE-D8 in the US alone costs $524. Adding a 900W power supply, two CPU coolers, and a case will cost more than $750 and all those parts have to be researched, ordered, assembled, wired, and configured. In Passmark baselines:

1. the highest CPU score for E5-2670 is 20855 on Levono D30,
2. 2nd is ASUS Z9PA-D8 at 20817, the Z9PA-D8 by the way is much less expensive- $333, than the Z9PE-D8 at $524,
3. 3rd is ASRock EPC602 at 20795,
4. 4th is Supermicro X9DRL-3F at 20697, and
5. 5th is Supermicro X9Dai at 20694. < that is the motherboard in the Supermicro Superworkstation SYS-7037A-i

While we're on this subject, as the Lenovo D30 perofrmas so well with the E5-2670, how about:

Computer PC Lenovo Thinkstation D30 Intel Xeon E5 Nvidia quadro K4000 Windows 7 > sold for EUR 300,00 ( Objectlocatie: Mestre, Italië)- the fastest, much less expensive, and could be up and running in a few hours. Even the Quadro K4000 is a decent GPU worth EUR 300 to sell so that system costs EUR 0!

Cheers,

BambiBoom



 

Spartan70NL

Reputable
Nov 5, 2014
22
0
4,510


Hi BambiBoom, that is exactly the reason why I wanted to go for a dual socket 2011 build. So many upgrade options available and because of the slightly/older gen cpu and ram prices should be pretty good. I think the Z9PE-D8 would be a great board to upgrade later on, it has many pcie and io ports without compromises. Price is higher than the other options put most lack in io ports like usb etc and while I looked at the Z9PA-D8 it did not have enough pcie slots either. Plus the Z9PE-D8 with its black pcb looks a lot better through the window of my 900D case :).

It's a price I'm willing to pay, maybe i'll upgrade cpu/ram/gpu maybe I won't but atleast I will have mobo supporting most. And I should be able to get hackintosh running on the Z9PE-D8. The specs of the Supermicro SYS-7037 series are amazing, and having lots of io and onboard audio, way overkill for me XD. About the K4000, I know the workstation GPU's are really expensive even the second hand lower class ones as the K2000. Don't understand why Quadro's don't dropp in price as much as the Xeons.