Building an 8-core, 3-GPU rig... advice?

[cyology]

So I'd like to build an 8-core computer with at least 3 GPUs in it, meaning I need a motherboard with two processor sockets and at least 3 PCI-e double wide slots in it. (This is for various scientific applications, the GPUs will be not be used for gaming so Crossfire/SLI support isn't necessary.) I've built computers before but I haven't done one in about 3 years so I'm not up-to-date on the latest rigs, and I need your advice!

I'm not particular about AMD or Intel. However, the only motherboard I've found that supports both multiple PCI-e slots and multiple processors is Intel's Skulltrail. This only seems to take Xeon processors, correct? If so, I would end up getting Xeon E5430s unless theres a better value for ~$500-600 that fits the socket 771.

Skulltrail MB: http://www.newegg.com/Product/Product.aspx?Item=N82E16813121330
Xeon E5430: http://www.newegg.com/Product/Product.aspx?Item=N82E16819117145

Some of the questions I have:
1. Does AMD offer anything comparable for 8-core multi-PCIe setups?
2. If I go with the Skulltrail+Xeon setup, what kind of case do I need? Any suggestions for a good one that will fit the MB?
3. When buying FB-DIMM memory which works in the board, what kind of differences am I looking at if I pick DDR2-667 vs DDR2-800 in terms of performance? Are there other performance characteristics I should look for in memory? Good manufacturers?
4. What kind of cooling do I need for this thing? I've never done anything other than stock heatsinks and fans, so I know essentially nothing about this... so links to Newegg items would be great.

Thanks for any help you can offer!

 

[xnem3s1sx]

Skulltrail isn't for Xeons, its for Core 2's, (but can run Xeons aparently) if you want Dual Xeons, you should get a server board, Skulltrail, I believe, has 4 PCI-e slots, but not enough room for GTX 280's (which are the best card for things such as folding@home as far as I'm aware) so you would need to liquid cool the computer, which would raise the cost considerably, (see http://www.dangerden.com/store/ as they are probably the only people with GTX or 4870x2 water blocks) I would recommend visiting a liquid cooling specific site for more in depth info on the subject.

 

[dagger]

1) AMD quad cpus perform below Intel equivalent, so typically, there is no reason to use AMD. However, there are AMD server motherboards that support 4 cpus and 4 pcie slots. So even though they're weaker individually, the extra 2 cpus might came in handy for your "scientific applications."
http://www.newegg.com/Product/Product.aspx?Item=N82E16813151085
Possible problems include 2 of the 4 pciex16 slots running at 4x each, the server motherboard's lack of overclocking bios options, and getting a case that fits the board (unlike all standard ATX cases for Skulltrail).

2) The Skulltrail will fit in any standard ATX midtower/full towers.

3) Generally, there is no noticeable performance difference with FB-DIMM, they just cost more. No ideas about manufacturers.

4) Get good cooling. Air cpu coolers are cheap compared to rest of what you wanted to get, and you can't have too much cooling.
http://www.frostytech.com/top5heatsinks.cfm

 

[cyology]

It seems to run a couple of Core 2s released in 771 packaging, which is normally the Xeon form factor. But the 771 Core 2s are all way out of my price range (~$1200 each) so I'd have to go with the Xeons. I can't find any 775 boards with two sockets.

Hmm, is there not enough room for 3 GTX 280s on the MB? In pictures it looks like the first 3 PCIe slots could accomodate double-wide GPUs, just not the last one... how can I tell how many GTX 280s a MB can support?

 

[dagger]

Skulltrail isn't for Xeons, its for Core 2's, (but can run Xeons aparently) if you want Dual Xeons, you should get a server board, Skulltrail, I believe, has 4 PCI-e slots, but not enough room for GTX 280's (which are the best card for things such as folding@home as far as I'm aware) so you would need to liquid cool the computer, which would raise the cost considerably, (see http://www.dangerden.com/store/ as they are probably the only people with GTX or 4870x2 water blocks) I would recommend visiting a liquid cooling specific site for more in depth info on the subject.

Skulltrail is originally designed for and released along with lga 771 qx9775.
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115044&Tpk=qx9775
It can also run cheaper lga771 quad Xeons. It will not fit normal lga775 Core 2s.

 

[dagger]

It seems to run a couple of Core 2s released in 771 packaging, which is normally the Xeon form factor. But the 771 Core 2s are all way out of my price range (~$1200 each) so I'd have to go with the Xeons. I can't find any 775 boards with two sockets.

Hmm, is there not enough room for 3 GTX 280s on the MB? In pictures it looks like the first 3 PCIe slots could accomodate double-wide GPUs, just not the last one... how can I tell how many GTX 280s a MB can support?

There is enough room for 3 dual slot gtx280, just not 4.

 

[cow_moo]

yeah

 

[skywalker9952]

Skulltrail isn't for Xeons, its for Core 2's, (but can run Xeons aparently) if you want Dual Xeons, you should get a server board, Skulltrail, I believe, has 4 PCI-e slots, but not enough room for GTX 280's (which are the best card for things such as folding@home as far as I'm aware) so you would need to liquid cool the computer, which would raise the cost considerably, (see http://www.dangerden.com/store/ as they are probably the only people with GTX or 4870x2 water blocks) I would recommend visiting a liquid cooling specific site for more in depth info on the subject.

Stop giving bad advice. Skulltrail is LGA 771 which is primarily a Xeon socket (until the introduction of the qx9775). Also it has rooms for three GTX 280s. GTX 280 = 2 slot video card. So it takes up the slot underneath it, if you'll notice there is a PCI slot under the first two PCIe x16 slots under the first two and a PCIe x16 slot under the third giving you a total of three GTX 280s. Why would you need to liquid cool the computer? You make no sense and what you say is misinformation, stop posting until you know more about what you are talking about. The OP was on the right track and hopefully is again.

 

[BigBurn]

Why not use 4870x2 instad of gtx280?

 

[cyology]

At the moment I'm actually getting a gtx280 and a 4870x2, due to the need to evaluate different programming APIs for GPGPU stuff. Eventually if one wins out (or a unified API emerges) I'd replace one of those with two of the winning ones. If ATI can get its API together, the 4870 hardware seems superior (more than double the number of stream processors, for example), but until then I need a nVidia card because CUDA is well-established and I know I can do what I need to do with it.

 

[faster3200]

Is there any option to wait? Core i7 should be out in November, that isn't too far away.

Also, yeah Skulltrial was basically meant to run with qx9775 procs but it can run with xeons.

 

[megamanx00]

Well that's kind of insane ^_^. Anyway for an 8 core setup having dual quad core Opterons would be better in some situations. It just depends on the applications your going to run on it, or write for it as the case may be. Seriously I barely have enough time to learn any APIs so I take it you're a full time student with too much time and a big student loan . I still haven't finished working with SM 3.0. So many things to do so little time .

 

[dagger]

At the moment I'm actually getting a gtx280 and a 4870x2, due to the need to evaluate different programming APIs for GPGPU stuff. Eventually if one wins out (or a unified API emerges) I'd replace one of those with two of the winning ones. If ATI can get its API together, the 4870 hardware seems superior (more than double the number of stream processors, for example), but until then I need a nVidia card because CUDA is well-established and I know I can do what I need to do with it.

That's not quite right, ATI stream processors and Nvidia ones are different. So you're comparing apples to oranges. Different implementations (CUDA/BrookGPU). ATI architecture has blocks of 5, each handling a different type of calculation. It's a bit hard to explain, but for all intended purposes, divide whatever number of stream processors in ATI gpus by 5, and you get a rough equivalent of Nvidia stream processor's realistic performance. Well, that's not actually right either, as the ATI architecture has a far higher potential in term of calculations, and it's only a matter of coding needed to harness it. Err... where were we? I got confused.

 

[cyology]

Is there any option to wait? Core i7 should be out in November, that isn't too far away.

Also, yeah Skulltrial was basically meant to run with qx9775 procs but it can run with xeons.

November is pushing it, I have some work to get done by early December, so it would have to be a darn good reason to try to develop on someone elses rig for all of October and half of November. I'm also not convinced Core i7 is a substantial upgrade for pure number crunching -- reduced cache size, same clock speeds. Hyperthreading isn't useful when cores aren't sitting idle very much.

Well that's kind of insane ^_^. Anyway for an 8 core setup having dual quad core Opterons would be better in some situations. It just depends on the applications your going to run on it, or write for it as the case may be. Seriously I barely have enough time to learn any APIs so I take it you're a full time student with too much time and a big student loan . I still haven't finished working with SM 3.0. So many things to do so little time .

Yeah I'm a PhD student with a grant for a honkin' computer. I don't know about having "too much time", and I'd love to only have to learn one API, but we don't seem to be there yet. I could just go with CUDA but I'm curious about the differences in architecture, as dagger mentioned -- its not entirely clear which one you can squeeze the most performance out of.

So I think I've settled on the Skulltrail system. What are the power supply requirements here -- do the processors have different connectors than the PCIe cards? I know each GPU needs two connectors, so I need a power supply with 2 CPU connectors, 6 GPU connectors and a connector each for the HD and DVD drive -- are these different types, what should I be looking for?

 

[dagger]

November is pushing it, I have some work to get done by early December, so it would have to be a darn good reason to try to develop on someone elses rig for all of October and half of November. I'm also not convinced Core i7 is a substantial upgrade for pure number crunching -- reduced cache size, same clock speeds. Hyperthreading isn't useful when cores aren't sitting idle very much.



Yeah I'm a PhD student with a grant for a honkin' computer. I don't know about having "too much time", and I'd love to only have to learn one API, but we don't seem to be there yet. I could just go with CUDA but I'm curious about the differences in architecture, as dagger mentioned -- its not entirely clear which one you can squeeze the most performance out of.

So I think I've settled on the Skulltrail system. What are the power supply requirements here -- do the processors have different connectors than the PCIe cards? I know each GPU needs two connectors, so I need a power supply with 2 CPU connectors, 6 GPU connectors and a connector each for the HD and DVD drive -- are these different types, what should I be looking for?

Looks like standard 24 pin plus 2 8 pins for 2 cpus. Don't worry about psu connectors, there are always adapters.
http://www.newegg.com/Product/Product.aspx?Item=N82E16812267004
http://www.newegg.com/Product/Product.aspx?Item=N82E16812887002

If wattage isn't enough, there are always expansion psus:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817153069

 

[cyology]

OK, so a PSU with the following specs:

1 x Main connector (20+4Pin)
1 x 12V(4+4Pin)
1 x 12V(8Pin)
8 x peripheral
8 x SATA
2 x Floppy
3 x PCI-E(6Pin)
3 x PCI-E(8Pin)

Would this support 2 Intel Xeons and 3 GPUs (each requiring a 6 pin and 8 pin) out of the box? Or do the Xeons take two of those 8 pins up? I'd like to make sure I order whatever adapters I need right away because I need to put in a single purchase order.

 

[cyology]

OK, so I guess you meant I needed 2x 12V (8pin) connectors, not an 8pin PCI-E connector for the CPU of course. I can only find a few PSUs managing that -- does this qualify?

1 x 24Pin motherboard connector (550mm)
2 x 8Pin EPS12V connector (750mm)
1 x 4Pin ATX12V connector (750mm)
2 x 8Pin PCI-E connector (550mm)
4 x 6Pin PCI-E connector (550mm)
2 x triple SATA connector (500mm +250mm + 250mm)
2 x triple 4Pin IDE & single floppy power connectors (500mm + 250mm + 250mm + 150mm)

I'm a bit unclear about the GPU power supply requirements: does a 4870x2 need a 8pin AND 6pin or just one or the other?

 

[taskforcetiny]

Damn! This level of building is way over my head lol!

Sounds awesome though, but I am curious as to what kind of scientific application you are using this for? Also does GPGPU stand for General Programming Graphical Processing Unit or something?

I can answer one of your questions though! Woot! The 4870 X2 requires both 8 and 6 pin connectors. A good PSU that I am using at the moment is the Thermaltake Toughpower 1200W PSU, and I'm running 3 X GTX280's off it. You should check it out:

http://www.thermaltake.com.au/Products/PowerSupply/W0133/Toughpower1200W.aspx

 

[dagger]

It would seem like the board doesn't actually require both 8pin connectors to be plugged in. So having the option to use psus with only 1 8pin should open up your choices considerably. In any case, there are surely adapters for that even if you end up needing the 2nd 8pin.
http://www.tomshardware.com/reviews/intel-skulltrail-part-1,1768-10.html

The rest of that overview also contain some useful information, read it.

 

[cyology]

Yeah, I would prefer to have those things fully powered (and I'm a little curious about overclocking) so I'll spring for one of the more expensive power supplies. The computer is shapin' up nicely, thanks for all the help guys. I'll post the final specs when I have em to make you jeal... err if you're curious.

For those interested in the scientific applications, they'll mostly be brain simulations and statistical analysis of brain imaging data, with the occasional computer vision style task thrown into the mix. GPGPU stands for general purpose graphical processing unit, meaning you can program the shaders and whatnot on these things to do arbitrary floating point computations rather than just those used for conventional 3D rendering. The newest GPUs are around 1TFlop peak performance with single precision, which is an order of magnitude or two higher than the CPUs we have, if you can program them correctly. But its much harder to program them due to their unique architecture (multiple types of memory, restrictions on what you can do, poor library support, etc.)

The reason I want the 8 CPUs still is because I do enough stuff that would benefit from the much simpler benefits of 8-way parallelism without having to restructure algorithms to run on shaders that its worth it.

 

[cyology]

Incidentally, I found out that the 4870x2s are essentially not supported by ATI for GPGPU applications right now, their API doesn't seem to recognize them as two cards and whatnot. I'm not liking their level of support for GPGPU stuff, so I'm probably just going to go with a single GTX280 for now and wait and see what happens over the next year or so. CUDA should keep me plenty busy.