Thanks for the response, my comments inline....
MRFS said:
Good for you! Just a few comments, because I can't confirm
from your photo these points (and forgive me for my lack
of complete information):
(1) PSU should intake cooler air from the fan grill in the bottom panel,
and exhaust out the rear; so, its intake fan should be pointing DOWN,
which is how you have it:
http://www.newegg.com/Product/Product.aspx?Item=N82E168...
>>>Yes, it's sucking in from beneath and blowing out the back.
(2) I'm not totally familiar with the Tesla's cooling: they should be
exhausting out the rear panel, like these:
http://www.nvidia.com/object/product_tesla_C2050_C2070_...
And, that's how you appear to have them installed.
So far, so good. Your experiment tells us a LOT!!
>>> That's right. The Tesla's have an internal intake fan which I assume blows
accross the chips and out the back. I've actuaklly removed the middle GPU you
see in the pic to improve air flow to the chip-set fan. This also improves air
flow to the remaining Tesla and the GTX.
(3) I would recommend adding 2 strong fans in the left-side panel:
one to force air into the gaps between your GPUs, and
one to force air into the space between the CPUs and upper GPU;
the ideal is to feed each heat source with cooler air, and
to exhaust that warmer air from each heat source without
warming any other interior components with that exhaust air;
>>>The case is actually the Cosmos "S" which is similar to the Cosmos but instead of
solid metal it's like a grill all over. The left side panel actualy has a big 8-9
inch fan built in. However, I removed it when we originally put the system
together because the CPU heatsinks were too big and obstructed the fan.
Last night I took a saw to the fan and hacked a few bits off so that I could
reinstall it today. I also added a fan in the base, as you suggested, which is
sucking air in from the floor and another in the front filling the lowermost three
drive bays, also sucking in.
(4) I'm also not totally familiar with that chassis: the photos at Newegg
show 2 exhaust fan grills in the top panel: you should have at least
one large fan installed there also, ideally 2 high CFM fans:
http://www.newegg.com/Product/Product.aspx?Item=N82E168...
>>>It's the Cosmos S and it has one fan in the top blowing out.
(5) if you have an empty 5.25" drive bay, I would add an intake fan there,
or at least remove one or more bay covers, in order to allow more cooler
air to enter at the front panel;
>>>See above
(6) all of the new fans that you add should have a variable speed
switch, to permit you to "tune" the speed of those fans;
a single-speed fan may not be fast enough at its single speed:
so, pay attention to CFM when you select these supplemental fans;
>>> hmmm... Well all the fans are the same so I hope that's going to be ok.
(7) make an effort to balance INTAKE CFM with EXHAUST CFM:
a wide variance will be less than ideal: you want constant,
even air flow around all interior heat sources, and the
exhaust from any given component(s) should not be
heating any other interior component(s);
>>> Thats a point! I have a front panel fan blowing on CPU1 which blows on CPU0
which in turn blows towards a rear panel fan blowing out. Potentially
sub-optimal? Maybe. But, CPU0 was never a problem so I'll ignore that for now.
(8) the photos at Newegg also show an optional intake fan
in the bottom panel: I don't see it in your photo, however:
http://www.newegg.com/Product/Product.aspx?Item=N82E168...
that's an excellent place to add an intake fan, because hot air rises
and cold air falls: thus, the air below the bottom panel should be
THE COLDEST AIR of any in contact with all 6 chassis sides.
>>> Yup! Added one today
I hope this helps.
>>> It all helps!
MRFS
So, to summarize, I've removed one GPU and added three fans. The result, the GPUs
are cooler when idle. When idle they're hanging around the 50C mark which is pretty
reasonable. I just a pretty heavy burn-in with all three (CUDA treats the GTX 295 as
two separate devices) running flat out for fifteen minutes and the highest they got
was 72C which is pretty good compared with 80C+ a few days ago.
The side fan seems to help a lot. The DIMMs were giving readings of about 70C before
and now are more like 55<65C, idle and working repectively.
On a more general note I'd add the following. This system is based on NVidias
recomendations for a "roll-your-own" Tesla personal supercomputer. They recommend
the Skulltrail for a three Tesla system with a cheapish Quadro card for graphics.
(Skulltrail has no on-board graphics). As you can see, we went for 2x Tesla and the
GTX 295. From personal experiance and talking to others in the past few days I'd say
that building a multi-Tesla system is no easy thing.
The problem is space and cooling, which is actually one problem. When I designed
this system the mobo question was whether to go with a single socket FoxCon
Destroyer or dual socket Skulltrail. The Destroyer has space for four Tesla, or
other large form factor GPUs. In either case you end up stacking them very, very
close together. This is bad.
We didn't have any problems for a long time because it's a test machine and untill
recently our tests have been small. We're now doing long intensive runs utilising
all GPUs and they get really hot.
If people can learn anything from this it is:.. If you build a multi-gpu system
cooling is going to be your problem. Despite NVidia's recomendations, don't stack
your GPUs so close together because they won't be able to breath and will generate
so much heat you might end up with problems elsewhere in the system like I did.
MRFS's suggestion to use a PCI mouted cooling solution is a good one. Stacking GPU
then slot fan, then GPU and so on would be smart thing to do but then where are you
gonna find a mobo with enough slots?