Sign in with
Sign up | Sign in
Your question
Solved

Oil submerged Servers

Last response: in Overclocking
Share
October 5, 2009 7:00:33 PM

I have several 8 core 8+ HD Servers running at my House, I'm a consultant and do a lot of VMware images for my different sandboxes, and I make the servers available to my clients at times, thus they need to be on 24/7.

That said the spare bedroom is LOUD as **** and rather warm, mainly warm because we keep the door shut because of the noise.

So my goal is not to overclock but to kill the noise, but you over clockers seem to be the experts in this area. <bows head to the experts>

I would like to know is how long will the MB, PS, and cables last submerged in say mineral oil, has anyone run one these longer than the http://www.pugetsystems.com/submerged.php#update5 system?

I wish I could afford the SSDs so they could be submerged, but that would be cost prohibitive for my data volumes.

So I think a hybrid approach would be my best approach, MB -Mother Board and PS Power Supply submerged and use blocks like http://www.frozencpu.com/products/7500/ex-blc-487/Koola... to cool the HD - Hard Drives.

I have experience working with Acrylic, and enjoy working with it as a hobby.

So anyone out there know of a system that has been running for years?





More about : oil submerged servers

October 5, 2009 7:34:00 PM

the only practical use for oil systems is to show off, the mess, leaks, upgrading, disposal, repairs,....

have you considered a window ac unit for the room or upgrading the loud parts for quieter ones

rack mount systems can run passive heatsinks...
m
0
l
October 5, 2009 7:51:50 PM

Those were my thoughts exactly about liquid cooling. My community won't allow a windows unit. They're all rack unit servers, I don't think I can passively cool them.

I'm not after show off, I have a real business use case here.

m
0
l
Related resources
October 5, 2009 8:24:22 PM

portable ac perhaps
m
0
l
October 5, 2009 8:28:24 PM

I think i've read on these forums a post about a system that was submerged for years in oil but could only run for about 12-14 hours before the whole tank of oil got too warm and started overheating and shutting down. Am not sure wheather such a system would even work. If the servers are on 24/7 and must maintain stability at all costs i would not recoment submerging them becuase this kind of cooling is just too new to base such systems on.

A room AC unit would be the best option in combination with heatsinks but i have no idea on the power consumption of AC units on 24/7 basis.

Good luck and keep us informed
m
0
l
October 5, 2009 8:36:37 PM

My goal is elimante the noise, my systems are running around 38C. I can mount a radiator in the AC closet for a heat exchanger. Adding a portable AC to the room will only create more noise, plus with an AC you have to beable to drain the condensate. This is a spare bedroom, I'd like to be able to have a guest sleep in it. There is no way anyone can sleep or be in that room.
m
0
l
October 5, 2009 8:46:48 PM

what about proper liquid cooling?
I know blocks would cost a lot but you dont sound like you cannot afford it.
m
0
l

Best solution

October 5, 2009 8:53:54 PM

As I understand it (i've done a bit of research in the past on this) the biggest thing with the mineral oil is that it breaks down over time. The heat issue with the hardware can be solved by either putting the system in a big enough tank of oil, and circulating the oil with a submersible pump, or by running the oil through a heat exchanger similar to the ones used by liquid coolers. The oil would need to be changed at regular intervals (with 24/7 operation, probably no more frequently than every 6 or so months). The bigges problem is that PSU's cannot be submerged, as the line voltages on the A/C side (120V) don't react well with the mineral oil (unless someone has found a way around that). HDD's are also non-submergible (as you noted) as are optical drives. That said, the cabling gets complicated, but not un-doable.

I've never seen this done with mineral oil on this scale, but there are high voltage power systems that use proprietary submerged cooling for extended periods. I think they use something called flourinert, which if I remember right, is about 300- 400 $ per gallon.

Any way you slice it, though you will still have the heat problem as the same amount of heat will be rejected, and it has to go somewhere. Without something to move it out of the bedroom (i.e. room A/C unit) it will still get warm in there.
Share
October 5, 2009 9:02:39 PM

The heat is not the issue, the AC closet is adjacent to the room. For the cost to block one system I can get the AC unit, and two systems I can build the closet.

I was trying to NOT block everything but the harddrives...I was just reading about the flourinet.
m
0
l
October 5, 2009 9:07:38 PM

Circulation would probably fix the problem but you can never be sure. I guess your buisiness is not something you want to risk. Submerging might work well or might fail epiccaly. As you said, a system (computer system we are talking of) of that scale has not been noted anywhere.
If you were some enthusiast who would build the submerged system on your sunday evening using the old hardware you have sitting there I'd surely encourage you BUT as i understand a large portion of what you do for living depends on this.

Also on a server if you leave out PSUs, hard drives and optical drives. this leaves out just the cpu to be cooled (and controller cards i'd guess).
I'd still suggest going for proper liquid cooling.
m
0
l
October 5, 2009 9:23:48 PM

I'm afraid you're correct, a hybrid system where the MB and its compnetns submerged and the Power supply would be optimal but I'm not going to risk it if no one has had success at running a submerged system for a long time.

Thanks for all of the input.
m
0
l
a b K Overclocking
October 5, 2009 10:27:56 PM

Why not just swap out fans? A bunch of YateLoon, Scythe Slipstream fans should do the trick.

Also a little FYI:
HDD blocks are not needed. EVER. They actually do more harm than good.

Budget?
m
0
l
October 5, 2009 11:03:26 PM

I have 4 1U, 1 3U, 1 4U or 5U servers.

The 3U has 8 hard drives, and the 5U has 12.

The 4 1U are IBM, and the other two are Supermicro.

AC Ductless AC installed 3K, plus materials for the room $750 (I can build the room).
This way I could add a few more servers.

So rationaly I think I need to come in way under $3750 for all 6 machines.

m
0
l
a b K Overclocking
October 6, 2009 12:29:09 AM

Mind listing more specs? esp. the CPUs and what your current set up (for cooling) is like?
m
0
l
October 6, 2009 3:03:08 AM

Lets do one that is really noisy...I used Belarc to pull this info:

Supermicro X7DB8

2.67 gigahertz Intel Xeon (2 installed)
16 kilobyte primary memory cache
2048 kilobyte secondary memory cache
64-bit ready
Multi-core (4 total)
Hyper-threaded (8 total)

Adaptec RAID 3805 Card
m
0
l
a c 86 K Overclocking
October 6, 2009 3:11:50 AM

Going AC is your best bet. Watercooling would be in the thousands, and you still have to remove the heat. Think of maintenance and upgrades, so much easier to use AC.

Your probably going to need a 3 ton AC unit (my house in Vegas has a 5 ton) It doesn't weigh 5 tons, but a 3 ton is pretty big still.

And the electric bill, whew!
m
0
l
a b K Overclocking
October 6, 2009 7:14:57 AM

as there servers they wont have there cpu's upgraded like ever, so why not try WC but instead of water in the loop id recommend ethanol this way a leak wouldn't fry your server, as for costs a duel 120mm rad for each cpu a block and a pump shouldnt cost any more then £200 per server. im not sure if you can buy water blocks for cpu's that fit a 1u server tho you might be able to use elbow joints
m
0
l
October 6, 2009 8:19:12 AM

a good water cooling setup would cost anywhere from 300-400 per server (not per processor) depending on the coolong capacity required (no. of procesors, usage etc). I really cant see any way to go over 3000 dollars for 6 servers. You can probably get away with 2000-2500 and still using the best parts in the market
m
0
l
a b K Overclocking
October 6, 2009 2:17:51 PM

You can buy an in room air conditioner at WalMart for about $500. It is free standing, plugs into a 110 outlet, and will do a fine job of keeping the room nice and cool. It has a dryer type hose coming out the back where it exhausts the hot air, that must be vented out a window is the only slightly tough thing about it. Other than that, you just wheel it in, plug it in, and you are set. They come with a vent attachment/adapter type deal that will fit into almost any type or size of window for the exhaust tube to hook onto.
http://www.stacksandstacks.com/windchaser-air-condition...
I have one similar to this to cool my house, and it works fantastic. I live in the northwest, so I don't need a whole lot of cooling, just something for the occasional hot summer day. I have about 1600 square feet, and it does a decent job. Won't freeze you to death by any means, but it keeps the whole house in the low 70's when it is in the upper 90's outside, and it is pretty easy on the electric bill. For a single hot room full of servers, this would be the ticket I would think.
Oil cooling has so many drawbacks, it is a novel idea at best reserved for those who want to do it, just to say they have done it. The mess, my god what a mess it will make. Can you imagine having to work on it? Can you live with stink of gallons of hot oil in your house?
m
0
l
October 6, 2009 5:40:43 PM

Forget oil unless you are willing to spend an inordinate amount of money and time setting up silicone seals and buying not just standard food-grade mineral oil, but actual electrical-grade mineral oil. The problem with the oil idea is

1. Your systems have to be submerged

a) Because of the high viscosity of oil, using cooling blocks that rely on impingement (or turbulence) for maximum heat transfer to the fluid in a cooling loop would be extremely difficult, as pumps that can do this would be louder than fans.

b) Only the mainboard and peripheral cards can be submerged. Hard drives and optical drives cannot be submerged.

c) Transformer-grade mineral oil is dielectric by nature, and has no problems with any electrical current, despite what a previous poster said. It has been used since the beginning of electrification to good effect as a heat transfer medium. So putting power supplies in oil should not be a problem.

2. The oil has to be circulated and cooled. Remember, oil is just a heat transfer medium. You still have to take the heat out of the oil in the end.

a) Because of oil's high viscosity (compared to water), you cannot use standard water cooling pumps and expect to get the same MTBF.

b) Any seals in the loop must be made from silicone. Mineral oil will break down carbon-based plastic or rubber seals.

c) Because of the lower flow rate of oil (see point 2a) you will need to use low flow radiators, which will not be as efficient as water-cooled high flow rate radiators.

In this situation, you could go oil, but you would be spending a lot of money on a solution that would, in essence, be a lot more difficult to maintain that going with water cooling. Especially if you are able to create a central cooling reservoir for all your systems and keep them in a closed loop, you would see a much higher return going with water cooling. Worried about leaks? Yes, they happen, but if you use deionized distilled water, it not the big catastrophe that folks here will worry you about. Deionized water does not conduct electricity.

IMHO, don't do oil unless you have experience and knowhow. Instead, you can plan a water loop. For the lines outside the systems, you can plan to use 3/4" PVC pipe. Have a tap at each box, one for water flowing in, and the other for water flowing out. You can purchase no-flow quick-style connectors for a reasonable amount of money and pull any single box off the loop. At the end of the loop, you have a single reservoir tied to a single large radiator and low speed high volume fan. You would still have the noise, but it would be the noise of a single large pump and a large fan. You still have to buy water blocks for all the processors, but you do not need to buy multiple pumps and reservoirs. And you might even be able to hire a local plumber to come in and help you with the build if you find that you don't have enough time. Some might even be willing to help you design such a system just for the experience.
m
0
l
a b K Overclocking
October 6, 2009 8:09:37 PM

Conumdrum said:
Going AC is your best bet. Watercooling would be in the thousands, and you still have to remove the heat. Think of maintenance and upgrades, so much easier to use AC.

Your probably going to need a 3 ton AC unit (my house in Vegas has a 5 ton) It doesn't weigh 5 tons, but a 3 ton is pretty big still.

And the electric bill, whew!

Either way, OP is lokking in to the $2k+ range with any solution. Imo, it's best to get water as it cost MUCH more for a AC down the road (aka electric bill).

@OP: What temps on the CPUs at 100% load (check when running Prime95)? Depending on the temps you are currently hitting, you MAY be able to get a fanless heatsink like this one: http://www.frozencpu.com/products/9212/cpu-dyn-34/Dynat...
I hear the veporchamber tech works quite well.

Quote:
I was just reading about the flourinet.

You have NO idea how much that stuff costs do you?
http://www.parallax-tech.com/fluorine.htm#price


At any rate, start by replacing the current fans with some Yate Loon or Scythe Slipstream fans.
m
0
l
October 6, 2009 8:38:24 PM

From what I gather, the OP has 2 problems.

1. Heat build up in the room.
2. Noisy fans in the computers

Easy fix.

1. Portable AC unit in the bedroom.
2. Replace noisy fans with quieter ones.

m
0
l
October 13, 2009 12:54:47 AM

Noise is the number one issue, if we could open the door the room would be a lot cooler.

Most server cases less that 4U don't have fans on the heat sinks, neither do any of my systems.

The loudest server has 16 hot swap HD bays, and 2 Xeon CPUs. These systems have extremely high volume fans, and this system has triple redundant power supply.

I'm getting quotes on AC for a closet in the garage, and going with water and SSD drives. I may also replace 4 of the older machines with one newer machine and virtualize it.

FYI I ping the guys over at Puget systems, and they report that the original system is still running strong.

Depending of how well the visualization goes with the SSD, I may Ebay the old and get new.

I appreciate everyone's input.
m
0
l
a b K Overclocking
October 13, 2009 10:34:19 AM

well im glad you have gone with our ideas it shows were not all stupi :D 
m
0
l
October 14, 2009 4:00:53 PM

I've got my home pc sitting in an acrylic tank with about 11.5 gallons of mineral oil. With no radiator cooling the oil will heat up enough to freeze the system. Adding a rad without fans will allow it to run close to max temp (62C) though I bet a heavy load would heat the oil too much again. Adding a large fan to the rad I can run the system just fine for normal use (movies/games etc..) and hardly go above 50C. For reference, my ambient temp is 38C when first turning the computer on.

No fan setup is almost silent. With the fan its not very loud either, but I can crank it up if needed.

System has been working since building it in early march 2009.
m
0
l
October 17, 2009 7:12:59 AM

You guys want to pick out the cooling blocks for this MB? P.S. I'll fill it with RAM.

SUPERMICRO MBD-X8DAH+-F-O Dual LGA 1366 Intel 5520 Extended ATX Dual Intel Xeon 5500
m
0
l
a c 86 K Overclocking
October 17, 2009 3:06:54 PM

Get two Swiftech GTZ CPU blocks. Any more cooling for the NB or mosfets is gonna be pretty hard to find since it's a very uncommon board for watercooling. Probably going to have to stick with fans for that. There is no reason to WC the ram.

It's a standard 1366 , so any cooler will fit, but you want something good, something proven, something easy.
m
0
l
a b K Overclocking
October 17, 2009 3:11:27 PM

^+1.
m
0
l
a c 86 K Overclocking
October 17, 2009 4:12:43 PM

I take back my recommendation for those CPU blocks. Now that your going to cool the server room why do you even need water? A good Air HS would be plenty and also MUCH cheaper. You won't be worried about noise anymore and the AC will hopefully keep the room in the 60's or even 50's, so water isn't needed.

Your looking at $100 or less to cool the Mobo on air vs $300+ on water. Your not overclocking, so you don't even need water at ALL.
m
0
l
a b K Overclocking
October 17, 2009 4:32:46 PM

^He didn't say he's going with AC. Just said he's getting quotes for it. Eitherway, he still wants a quiet PC, so going WCing he may be able to passively cool it (or use very few fans?)?
m
0
l
a c 86 K Overclocking
October 17, 2009 5:13:11 PM

If it's in a seperate room then quiet isn't an issue. An AC unit isn't that quiet anyway.
m
0
l
a b K Overclocking
October 17, 2009 6:11:22 PM

^Good point. But anyways, will the OP be able to go with passive cooling on the rads(320/480) with an AC?
m
0
l
a c 86 K Overclocking
October 17, 2009 6:39:48 PM

I'll try to make this more clear. Why does he need any rads at all when he's got cool air across the rigs? He's not looking to overclock either. He can go air cooled heatsinks for much much less money. He originally wanted to go oil or water because of the heat buildup. Once he understood the heat isn't going to be any less in the room with oil or water cooling, he decided to look into AC.

Since he's looking at a seperate room for housing the servers, why worry about the noise?
m
0
l
a b K Overclocking
October 17, 2009 8:52:29 PM

^ I see what you mean now, if OP's housing it in a different room, noise shouldn't be an issue and therefor no need for WCing.
m
0
l
October 18, 2009 3:41:07 AM

I would totally forget about submerging servers in oil. It's messy and too much of a hassle when you have to do repairs or want to upgrade. The oil will still get hot and cause problems if it gets too warm. So you end up cooling the oil instead of the servers, it just adds another annoying layer to the problem.

Water cooling is probably not a good choice either, it's kinda expensive and something will eventually leak. I know someone is going to reply that my water cooling never leaks and it been running for blah, blah, blah...... Something will always fail eventually when you have enough equipment and have it running 24/7 for years. I wouldn't risk my business on a line leaking at the top of the rack and possibly taking out every thing underneath it. Which will conveniently happen when you are out of town.

I have a separate air conditioned server room at the office so I don't care about noise, but at home I replaced everything with Scythe S-Flex fans because I hated the hair dryer level sound coming from the old fans. There is still a low hum, but still quieter than the aquarium that's in the same room.

Liquid bearing fans(like Scythe S-Flex) will be your best bet. At least on NewEgg, I couldn't find anything smaller than 80mm which could be too big. The noisiest fans I found are those cheap 40mm whiny fans manufactures use. Most of the time they are at the end of the thin case blowing/pushing air. So they won't fit inside, but maybe mounting them on the outside would work. Anyone ever try that?



m
0
l
a b K Overclocking
October 18, 2009 3:48:57 PM

Quote:
fail eventually when you have enough equipment and have it running 24/7 for years

You NEVER run an WCed system for more than 6 -12 months with out re-doing the loop.
m
0
l
a c 86 K Overclocking
October 18, 2009 11:06:59 PM

Peeps do, some get away with it, some don't.

Most times WC is a hobby, so the 6 month drain is fun and part of having a perfect running loop.

Some just build and forget. It's their stuff, no worries. The plus to peeps like that is reading the fun posts.
m
0
l
October 19, 2009 4:20:17 PM

I've had a loop running for nearly 2 years now without having to drain the loop once. All I've had to do is add fluid lost due to osmotic loss through the PVC tubing. Granted, I used Fluid XP+ when I filled the loop last (mostly to test the stuff out) and it has been biotic-free since it was first filled. I guess I could drain it just to see if there is any biotic growth at all, but it looks as clear in the tube as the day I first bled the air out two years ago.

As far as reducing overall noise, if you don't want to go though the hassle and expense of oil or water cooling, then air cooling with quieter fans is the only viable option.
m
0
l
!