Intel’s 24-Core, 14-Drive Modular Server Reviewed

Main And I/O Cooling Modules

The rear mounted Main Cooling Modules have two fans inside each casing that pulls air in from the front of the chassis and out the back.  The rear mounted Main Cooling Modules have two fans inside each casing that pulls air in from the front of the chassis and out the back.

Cooling is a key factor when running the MFSYS25. About 80% of the MFSYS25 front panel is perforated, so if you put your hand in front of the running chassis, you can feel the amount of air drawn in from different parts of the panel. There are three main sources for this induction.

In the rear of the main chassis are two hot-swappable main cooling modules that are used to pull air through the heavily perforated front panels in front of the compute modules. These large modules sit in a vertical arrangement so that each main cooling module is positioned to cover either the top three compute modules or the bottom three.

As the fans spin, air rushes over each of the compute modules’ voltage regulator, CPU, and memory banks before exiting through the main cooling module. Each of these two larger cooling devices consists of two 4.5" fans in a 1+1 redundancy configuration and sits in a tandem position. Each fan then plugs into a small circuit board inside the module housing, which in turn plugs into the MFSYS25 midplane.

As mentioned before, each of the main cooling modules cools down three of the six compute modules, so if there is a problem with a faulty fan, it should be replaced immediately. The user manual states that hot-swappable cooling devices should be replaced within a minute, especially if the MFSYS25 is fully populated with six compute modules.

The smaller I/O cooling module sits on the front of the main chassis, just below the storage enclosure.  Inside this smaller module are six 1.5" fans that run in a 3+3 redundancy configuration. The I/O module pulls in air from the front of the chassis and directs the flow to the back, which cools the I/O devices in the rear of the system. Like the other two cooling modules, the I/O cooling module is hot-swappable as well.

Placed in front of the MFSYS25 chassis, the I/O Cooling Modules keep the Modular Server’s Management Modules from over-heating.Placed in front of the MFSYS25 chassis, the I/O Cooling Modules keep the Modular Server’s Management Modules from over-heating.

One interesting observation I made during this evaluation was the behavior of the entire cooling system when one of the three modules was pulled out. When the chassis loses a cooling component, the remaining two devices go into high gear. From the increase of air flow, it would seem that the chassis is compensating for the missing device. However, neither the MFSYS25/MFSYS35’s user guide nor the technical product specification manual mentions this behavior, while an email from an Intel representative said that this is normal behavior.

I found out that in case of a cooling device failure that the remaining main cooling module will continue to run at 100% activity and can run in this state for a while.  On the other hand, if there was a problem with the sole I/O cooling module, then an immediate replacement would be necessary.

I couldn’t wait to test the overall heat coming out of the MFSYS25 at different compute module configurations.  After bringing up all three compute modules (one at a time), I sequentially shut them down until I was just running the chassis by itself. The startup and shutdown of the compute modules were completed in 15-minute intervals.

I started with just the chassis and the disk drives powered on. Note that the compute modules were in their bays the whole time, but no command to start them up was issued until later.

I then powered on the compute modules in 15 minute intervals. By interval 4, you can see that the amount of heat coming out of the main cooling modules increases dramatically.

As might be expected, when power consumption increased and after I added more compute modules to the chassis, there was a significant increase in heat, not only from the newly powered-up components, but from the power supply modules as well.  There’s at least a 19 degrees Fahrenheit heat increase coming from the cooling modules as they pull the hot air away from the three compute modules. If I had loaded the chassis with three more compute modules, then I’m sure the line representing the main cooling modules would be close to the power module’s temperatures. At interval 5, I started shutting down the compute modules at 15 minute intervals. This “shutdown” test produced an almost mirror-image of the system as everything was booting up during the first half. Note that temperature coming from the compute modules’ front panel only went up and down by three degrees. This just means that the CPUs near the front of the compute modules were probably being well ventilated by the flow of fresh air coming from the front of the chassis.

Create a new thread in the US Reviews comments forum about this subject
This thread is closed for comments
39 comments
    Your comment
  • kevikom
    This is not a new concept. HP & IBM already have Blade servers. HP has one that is 6U and is modular. You can put up to 64 cores in it. Maybe Tom's could compare all of the blade chassis.
    4
  • kevikom
    Also I did not see any pricing on this. Did i miss it somewhere???
    4
  • sepuko
    Are the blades in IBM's and HP's solutions having to carry hard drives to operate? Or are you talking of certain model or what are you talking about anyway I'm lost in your general comparison. "They are not new cause those guys have had something similar/the concept is old."
    0
  • Anonymous
    Why isn't the poor network performance addressed as a con? No GigE interface should be producing results at FastE levels, ever.
    0
  • nukemaster
    So, When you gonna start folding on it :p

    Did you contact Intel about that network thing. There network cards are normally top end. That has to be a bug.

    You should have tried to render 3d images on it. It should be able to flex some muscles there.
    1
  • MonsterCookie
    Now frankly, this is NOT a computational server, and i would bet 30% of the price of this thing, that the product will be way overpriced and one could buid the same thing from normal 1U servers, like Supermicro 1U Twin.
    The nodes themselves are fine, because the CPU-s are fast. The problem is the build in Gigabit LAN, which is jut too slow (neither the troughput nor the latency of the GLan was not ment for these pourposes).
    In a real cumputational server the CPU-s should be directly interconnected with something like Hyper-Transport, or the separate nodes should communicate trough build-in Infiniband cards. The MINIMUM nowadays for a computational cluster would be 10G LAN buid in, and some software tool which can reduce the TCP/IP overhead and decrease the latency.
    2
  • Anonymous
    less its a typo the bench marked older AMD opterons. the AMD opteron 200s are based off the 939 socket(i think) which is ddr1 ecc. so no way would it stack up to the intel.
    1
  • Anonymous
    The server could be used as a Oracle RAC cluster. But as noted you really want better interconnects than 1gb Ethernet. And I suspect from the setup it makes a fare VM engine.
    0
  • ghyatt
    I priced a full chassis out for a client, and it was under 20k...
    1
  • navvara
    It can't be under 20K.

    I reallty want to know what the price of this server is.
    0
  • Anonymous
    Actually it is under $20k for a fully confgured system with 6 blades - i priced it up online. You can push it a bit higher than this if you go for very high end memory (16GB+) and top bin processors but for most the fully loaded config would come in around $20k. It's very well priced.
    0
  • kittle
    The chassis for your client was under 20k... np

    but to get one IDENTICAL to what was tested in this article - whats that cost? I would think "price as tested" would be a standard data point.

    Also - the disk i/o graphs are way to small to read without a lot of extra mouse clicks, and even then i get "error on page" when trying to see the full rez version. Makes all that work you spent gathering the disk benchmarks rather useless if people cant read them.
    1
  • Anonymous
    the price as tested in the article is way less than $20k. they only had 3 compute modules and non redundant SAN or Switch. Their configuration would cost around $15k - seriously just go and price it up online - search MFSSYS25 and MFS5000SI
    0
  • asburye
    I have one sitting here on my desk with 6 compute modules, 2 Ethernet switches, 2 Controller modules, 4 power supplies, and 14-143GB/10k SAS drives. The 6 compute modules all have 16GB RAM, 2-Xeon 5420's each and 4 of them have the extra HBA card as well, our price was < $25,000 with everything including shipping and tax. The Shared LUN Key is about $400. We bought ours about 2 months ago.
    3
  • Shadow703793
    nukemasterSo, When you gonna start folding on it Did you contact Intel about that network thing. There network cards are normally top end. That has to be a bug.You should have tried to render 3d images on it. It should be able to flex some muscles there.

    Nahhh... you don't run F@H on CPUs any more ;)
    You run it on GPUs! CUDA FTW! :P
    0
  • nukemaster
    yeah, thats true. CUDA kills in folding.
    0
  • Anonymous
    Thanks for the comments/suggestions/questions everyone. Your input is appreciated and will be applied to future reviews.

    We're addressing the issue with the network test results. - julio
    0
  • Area51
    This is the only solution that I can think of that has the integrated SAN solution. None of the OEM's (Dell, HP, IBM) can do that in their solution as of yet. Also if you configure the CPU's with L5430's this becomes the perfect VMware box.
    As far as power switch... Remember that in a datacenter enviornment you do not turn off the chassis. there is less chance of accidental shutdown if there is no power switch on the main circuit. This is 6 servers network switch, and a SAn solution in-one you do not want a kill switch. that is why no OEM ever puts a power switch in thier blade solution Chassis.
    2
  • Anonymous
    Hi folks. We're re-running the network test this weekend. Stay-tune for the update. - julio
    1
  • MonsterCookie
    I went over the review/test, and as far as i understood in this system there is only a single GLAN switch.
    Besides, the individual nodes do not have their own HDD, but a NAS instead.

    This is particularly bad, because the disk-I/O is also handled by this poor LAN switch.
    One should use at least two switches: one for internode communication, and the other for NFS and so on.


    Second point: if those prices which i have seen in the forum are right, than 20k$ is equivalent to 15600Euros.
    For that money i can buy from the company where we are buying our equipment from the same system build from Supermicro 1U Twins. For this price i even have Dual GLAN per node, and one Infiniband per node.
    This system above could be called indeed a computational server, and the Intel system is just something like a custom made weakly coupled network of computers which is coming with a hefty price tag.
    Of course one could argue that buying 3 Supermicro twins plus an Infiniband switch is not as neet looking as this Intel, but once it is in a rack, who cares?
    I would not really would like to listen to this intel machine on my desk anyway, so that should be put in a nice rack as well.
    0