Running four 9800GX2 cards is possible

[kjoost]

I am a researcher at the University of Antwerp in Belgium. We have recently developed a "superPC" (made from consumer hardware) that runs eight GPUs on four NVIDIA 9800GX2 cards.

Given the controversies about how much power would be needed for such a system, if aircooling is sufficient, etc., I would like to share our experiences:

http://fastra.ua.ac.be

Has anyone else built such a system too?

 

[aevm]

Wow!!!

Is it hard to write software with CUDA? Just curious...

 

[kjoost]

Wow!!!

Is it hard to write software with CUDA? Just curious...

The CUDA programming language looks a lot like C. If you know how to program in C, it is quite easy to learn to write CUDA programs. However, getting the high speedups requires a lot more work. The order in which your program accesses the various types of memory are of great influence to performance.

 

[spanner_razor]

It's a crossfire motherboard though, did you have to write custom drivers to allow both motherboards and gpus to work with 8 gpus.
Isn't gpu overheating a problem too, is that why the side of the case is off as it's too hot with it on.

 

[aevm]

Impressive. I hope your solution is copied all over the world and helps medicine advance a little faster.

I loved the section about comments from forums. So you went ahead and did it anyway, after everybody said it was impossible

 

[aevm]

It's a crossfire motherboard though, did you have to write custom drivers to allow both motherboards and gpus to work with 8 gpus.
Isn't gpu overheating a problem too, is that why the side of the case is off as it's too hot with it on.

Yeah, they explain it on the Web page. They get 86 degrees Celsius under load. This will make the cards die faster than usual, true, but by that time they'll just get some 9900GX2 cards instead or whatever.

 

[kjoost]

It's a crossfire motherboard though, did you have to write custom drivers to allow both motherboards and gpus to work with 8 gpus.
Isn't gpu overheating a problem too, is that why the side of the case is off as it's too hot with it on.

Yes, it is a crossfire motherboard. The reason why we opted for a crossfire motherboard is that, as far as I know, no SLI motherboard exists with four PCI-Express slots and dual slot spacing between the slots.

However, for our computations we don't need SLI: see the technical FAQ http://fastra.ua.ac.be/en/technicalfaq.html for more explanation.

 

[aznguy0028]

wow. awesome

1600w PSU!

 

[Guest]

always cool seeing posts like this.

 

[bobwya]

I am a researcher at the University of Antwerp in Belgium. We have recently developed a "superPC" (made from consumer hardware) that runs eight GPUs on four NVIDIA 9800GX2 cards.

Given the controversies about how much power would be needed for such a system, if aircooling is sufficient, etc., I would like to share our experiences:

http://fastra.ua.ac.be

Has anyone else built such a system too?

It would have been nice to see some butt-kicking watercooling there!! (You would have to dissipate 550 Watts + from those GPUs in a single loop - ouch !!) If the flowrate could be kept up then the stability and OC from some good GX2 waterblocks would be sweet...

http://www.dangerden.com/store/product.php?productid=336&cat=48&page=1
http://ekwaterblocks.com/shop/product_info.php?cPath=21_31_42&products_id=254

You would at least need a Liang DDC Ultra pump before and after the GPU blocks in series in a seperate/parallel loop with dual PA120.3 rads. or similar!! I think the restriction of the GPU blocks would cause the water to heat up too much in each GX2 block however... Would be interesting to see such a setup up and running though!!

Uhhmmm OK maybe aircooling is easier after all!! (Certainly would be tidier once you start housing all those pumps and radiators required in such a setup...)

BTW I am sure you could easily hotwire a Ultra-X3 1600Watt PSU with 2x extra PCIe-8 connectors as it is also modular (the Thermaltake PSU) and has 1400 Watts on the (single) 12V line!!

Just out of instance would the FSB be a limiting factor in the CPU if you had used an Intel quad core CPU? Since the CPU appears to act as kindof a crossbar... I presume you went with AMD because of hypertransport?? (Nehalem next year, eh? LOL)

Really nice setup... Nice to see CUDA being used so effectively!! Makes up for the usual **** posts on THG!! LOL at kicking the butt of an expensive multiprocessor x86/x64 CPU -based Supercomputer!!

Bob

 

[chookman]

Thats awesome well done. Impressive read

 

[V3NOM]

wow, cool to resurrect this thread now, four 4870X2's!!!!!!!!!! ))))))))

 

[V3NOM]

no, seriously... four 4870X2's would be like 120 degrees, but HELL! it would work on that crossfire board! nice job by the way, i'm hoping that quad-card, octo-core system is still alive? not melted or anything?

 

[aevm]

Just curious, do ATI's drivers support 8 GPUs? I wonder how Crysis would work on a setup like that...

 

[Grimmy]

Thought this thread was familiar:

4x9800GX2: will it overheat?

 

[rubix_1011]

You would at least need a Liang DDC Ultra pump before and after the GPUgpu blocks in series in a seperate/parallel loop with dual PA120.3 rads. or similar!! I think the restriction of the GPU blocks would cause the water to heat up too much in each GX2 block however... Would be interesting to see such a setup up and running though!!

No, wrong. You would likely need separate loops for GPUs and CPU(s), but aside from having a good pump and a lot of radiator space, it shouldn't be a problem.

Honestly, stop thinking that watercooling means getting super heated water from your GPUs and melting everything. Watercooling doesn't allow components to heat up nearly as much as air cooling, so your 85C GPU is only getting 40C under load with water. As long as you have sufficient heat exchanger surface area and good fans, you can displace the heat from the water and allow the loop to maintain very cool temps vs what you would see with them staying air cooled. Apparently, the guys building these have a pretty good budget, so if they wanted, it is probably a better solution than adding 200 fans to a case.

 

[Grimmy]

200 fans??

Say why not perhaps 4 of these fans?

Commonwealth 172x150x51mm Dual Ball Bearing AC Fan

Should help keep the bugs out with 230CFM.

. <(what he say?)

I SAID IT SHOULD KEEP THE BUGS OUT..

 

[aevm]

LOL. How about this:
http://www.walmart.com/catalog/product.do?product_id=9107780

 

[invisik]

Very interesting! keep me posted about these type of projects you guys do.
One quick question in this situation wouldn't using a server cpu be more effective?

 

[V3NOM]

Just curious, do ATI's drivers support 8 GPUs? I wonder how Crysis would work on a setup like that...

err yeah slight problem being a four-slot crossfire bridge...

 

[V3NOM]

meh, they will probably just make their own drivers again kudos to them!

 

[jaragon13]

I am a researcher at the University of Antwerp in Belgium. We have recently developed a "superPC" (made from consumer hardware) that runs eight GPUs on four NVIDIA 9800GX2 cards.

Given the controversies about how much power would be needed for such a system, if aircooling is sufficient, etc., I would like to share our experiences:

http://fastra.ua.ac.be

Has anyone else built such a system too?

Nope.I don't have ~10,000 dollars to spend on power calculators.

Would be awesome,though.