CUDA Workstation

[atomiktoaster]

I'm working with an engineering team on a custom CUDA-based simulation tool. We'll be buying a couple systems to run it, but the software details aren't all worked out yet. Assuming that the software will be able to use all the GPU memory bandwidth we can throw at it, I'd want to put 3 GTX 280s or 285s in the box, with plenty of storage for the archived results and some fast storage to hold the OS and working results. It also has to sit next to my desk all day, so I want to keep it quiet. Price isn't a huge issue, as long as it's cheaper than whatever Dell I could get with comparable performance.

Here's what I'm thinking so far:

Core i7 920, moderately OCed

ASUS P6T6 WS Revolution LGA 1366 Intel X58 ATX Intel Motherboard
http://www.newegg.com/Product/Product.aspx?Item=N82E16813131358

3X BFG Tech BFGEGTX2801024H2OCWE GeForce GTX 280 1GB
http://www.newegg.com/Product/Product.aspx?Item=N82E16814143143
For CUDA only. I need to push the memory clock on these as far as I can. Water cooling is for noise mainly, as well as heat. I'd be happy to go with a GTX 285 or a brand like EVGA if they sell one with a waterblock.

PNY VCQFX580-PCIE-PB Quadro FX580 512MB
http://www.newegg.com/Product/Product.aspx?Item=N82E16814133273
To actually run the monitors. Quadro because we use PRO/E pretty heavily

CORSAIR CMPSU-1000HX 1000W
http://www.newegg.com/Product/Product.aspx?Item=N82E16817139007

2X CORSAIR XMS3 6GB (3 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600
http://www.newegg.com/Product/Product.aspx?Item=N82E16820145242

4X Western Digital Caviar Black WD1001FALS 1TB 7200 RPM in RAID 10
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136284
Archive for results.

2X Intel X25-M SSDSA2MH080G1 80GB
http://www.newegg.com/Product/Product.aspx?Item=N82E16820167005
Either split for OS/apps and storage of working results, or or combined for both uses in RAID 0

SYBA CL-HD-MRDU25S Mobile Rack for 2.5" SATA
http://www.newegg.com/Product/Product.aspx?Item=N82E16817998038

HP Black 22X DVD+R 12X DVD-RAM SATA
http://www.newegg.com/Product/Product.aspx?Item=N82E16827140040

I've built a couple computers before, but never in this price range (usually <$600), and never with an i7, RAID, SSD, multi-GPU or water-cooled setup. I could use some suggestions on a case and water cooling setup, as well as any thoughts on the build as a whole.

 

[kubes]

I guess why go with 3x280's or 3x285's when really your looking to do workstation type work. If money really isn't much of a concern maybe you should look into the Nvida Tesla Gpu option or Nvidia Quadro. These are designed much more optimized for what it seems your looking to do than gtx 200 series cards. But of course there's a much heftier price tag on these cards.

 

[atomiktoaster]

The bottleneck for us is memory bandwidth, which is just the product of the memory clock and bus width. The Teslas have the same 512-bit bus width, but they ship at 1600 MHz (DDR) memory clock. The 280's have a 2450 MHz memory clock. Plus, they're probably 80% less in terms of cost. I know the Quadros have the drivers you need for workstation tasks, which is why I'm including one to actually run my monitors. However, we're writing the main application software ourselves to run on gaming hardware, since it's usually several months of development ahead of workstation parts, and much much cheaper. $1000 or $1500 in GPUs for 3-5 machines is doable, but $8000 in GPUs for each machine is going to hold us back. Plus, I'm not sure anyone sells water-cooled tesla or quadro cards.

 

[atomiktoaster]

I did some more research, and I've found some a build that's along the lines of what I had in mind.

http://fastra.ua.ac.be/en/index.html

It'd be pretty simple to copy this parts list and replace the 9800GX2s with GTX 295s or GTX285s, but I'll definitely need a lot of storage for my results. Assuming that I can make use of additional GPUs as effectively as the FASTRA team, what sort of case and cooling situation do I use to keep that many GPUs reasonably quiet and less than deafening?

This motherboard should be able to fit a 4th GPU.

ASUS P6T7 WS Supercomputer
http://www.newegg.com/Product/Product.aspx?Item=N82E16813131390

This might be another 4 GPU option
ASRock X58 Supercomputer
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157150

Do any of the X58 motherboards have a molex socket for additional PCIe slot power (75W per slot) like the MSI K9A2 Platinum had? That was an issue with a 4 X 295 build posted on the NVIDIA forum using a Foxconn destroyer board.

The Corsair PSU might be able to take 3 285s, but 4 295s is going to require a beast like the 1500W Thermaltake that FASTRA used, or two PSUs.

 

[kubes]

You won't be able to hook up 4 x 295's. The most you can do is two ( = four gpu cores) and a dedicated physics card (which you don't really need).

If you wanted to do two sli'd setups in the same rig where you have 4 x 295's, then ur software would somehow have to figure out how to bridge them together. I'm not even sure if that's possible though.

Even though your memmory bandwith strapped ...doesn't the workstations cards optimize on this? I guess i'm still confused why your going down the gtx 295 route and not a workstation card. Maybe this isn't so much a hardware issue but a software issue?