Sign in with
Sign up | Sign in
Three Xeon E5 Server Systems From Intel, Tyan, And Supermicro
By ,
1. Three 2P Xeon E5-2600 Platforms Compared: Intel, Supermicro, And Tyan

Recently, Chris Angelini wrote about the Xeon E5-2687W CPUs in Intel Xeon E5-2600: Doing Damage With Two Eight-Core CPUs. Chris was able to compare Intel's flagship model, with a 150 W TDP, to the two generations that preceded it, along with a Core i7-3960X on the desktop. Around the same time, we were trying to get our hands on a number of barebones server systems to compare against each other.

We ended up with three very different models from Supermicro, Tyan, and Intel, along with a pair of Xeon E5-2690 processors. The -2960s are the highest-end Sandy Bridge-EP-based chips available for the server market (Chris' -2687Ws are workstation-specific). All three systems occupy 2U of rack space and were designed to accommodate 135 W CPUs, along with tons of memory and several hard drives.

Although much of our attention is spent on enthusiast-oriented desktop hardware and mobile technology, there's unquestionably an allure to a 16-core, 32-thread machine (cooled with passive heat sinks) sporting up to 768 GB of memory, redundant high-efficiency power supplies, hot-swap SAS bays, and all of the other features we associate with higher-end servers. The features found in this space are often precursors to capabilities that trickle down to the desktop, after all.

Three Intel Xeon E5-2600 2U Barebones SystemsThree Intel Xeon E5-2600 2U Barebones Systems

Two of the three platforms in this round-up employ custom-designed motherboards with accompanying 2U enclosures built to fit them. This type of proprietary design is common in the server segment, and it represents a significant departure from the desktop world where proprietary designs are often shunned by builders in favor of standard form factors and the interoperability benefits they bring. In servers, however, more custom layouts allow vendors to optimize cable routing and cooling, optimize expansion capabilities, and lower power consumption.

Again, the three vendors participating in this round-up are Supermicro, Intel, and Tyan. Gigabyte and Asus were invited as well, but weren't able to provide samples in time for our story. Intel provided two of its newest Xeon E5-2690 CPUs to serve as the test chips in this review. The Xeon E5-2690 is the highest-end non-workstation E5-2600-series CPU Intel sells, and it is rated at a 135 W TDP. Because 2U enclosures do pose thermal challenges, which in turn can impact power consumption, these are ideal test mules for this story. And we wanted to use barebones systems for our evaluation because they function as a foundation for servers, and are meant to be populated with a range of CPUs, memory, storage, and expansion cards.

Today, we're going to be looking at the hardware contained within each platform, along with its remote management capabilities. Although manufacturers don't release exact numbers on the number of systems sold with remote management and KVM-over-IP functionality, informal estimates point to 70% or more of these types of systems including remote management features. It is clearly an important consideration, and something Intel has put emphasis on with its enterprise desktop vPro initiative. Once we look at the systems themselves, we'll have a look at warranty and support options, the relative performance of each system, power consumption, and acoustics.

2. The Rules, Contenders, And Test Setup

We set some ground rules for the vendors that wanted to participate in this story. The manufacturers received instruction that all platforms were to have:

  • At least two LGA 2011 interfaces supporting at least 135 W CPUs to accommodate the Xeon E5-2690 processors we planned to use.
  • At least two gigabit Ethernet ports (excluding any dedicated IPMI NIC.) Most server motherboards already include at least two gigabit ports. Although some vendors offer solutions with four or more on-board ports, we wanted the option for at least two.
  • IPMI or similar remote monitoring and KVM-over-IP integrated or through an included add-in module. Nobody has industry-wide numbers breaking out how many servers have IPMI 2.0 and remote KVM-over-IP built-in versus provided by an external controller, though we've heard that as many as 70% of these platforms ship with these features.
  • No additional SAS controller (such as an value-added LSI SAS2008). However, PCH-enabled SAS is OK. One of the major new features with the Patsburg chipset is its ability to enable up to eight SAS 3Gb/s ports using an on-die Storage Controller Unit. This is really an aggressive move by Intel, as LSI sells many of its own controllers that augment Intel-based servers. If Intel can achieve parity with LSI in terms of features, then this will become a major industry trend.
  • No built-in InfiniBand or 10 GbE, as typical servers in this segment do not have those more enterprise-oriented extras as integrated features. Vendors do offer solutions with them, but in lower volumes.
  • A price, including any required accessories, below $1800. This figure may seem high, but it is actually very close to average for a barebones chassis, motherboard, and redundant power supplies in the segment we're targeting.


Generally, most vendors complied with our list, which made our comparison easier. There were a few exceptions, which we'll note as we go through the story. Notice that we didn't ask the manufacturers to adhere to a size restriction, since we wanted to make sure that vendors had the flexibility to submit different types of servers. As the picture on the previous page reveals, all three vendors submitted 2U chassis. That's a pretty telling indicator of what the industry considers its sweet spot.

The other key piece of this puzzle is the processors we're using. Intel sent two Xeon E5-2690 CPUs, which are very similar to the workstation E5-2687W parts reviewed previously (except that 200 MHz of base clock rate is sacrificed for a 15 W-lower TDP, a trade-off that makes sense in a cramped 2U enclosure). The Sandy Bridge-EP-based Xeon E5 platform's new features were covered in this short video, which is worth watching as an overview.

Tom's Hardware Talks To Intel Architect Ken Creta

Another important consideration was our choice of memory. Trying to be vendor-agnostic, Micron, Kingston, and Samsung memory were all used with each barebones system. Also, the systems were tested with both unbuffered ECC and registered DIMMs. After verifying that compatibility was uniform across each system, we settled on registered DIMMs for testing, since they're most common in dual-processor servers.

Test Hardware
Processors
2 x Intel Xeon E5-2690W (Sandy Bridge-EP) 2.9 GHz, Eight Cores, LGA 2011, 8 GT/s QPI, 20 MB Shared L3, Hyper-Threading enabled, Power-savings enabled
Barebones
Supermicro 6027R-N3RF4+

Tyan GN70-K7053 (S7053 + KGN70M1)

Intel R2208GZ4GC "Grizzly Pass"
Memory
Micron 64 GB (16 x 4 GB) DDR3-1333CL9 Registered ECC

Samsung 64 GB (16 x 4 GB) DDR3-1333CL9 Registered ECC

Kingston 32 GB (16 x 4 GB) DDR3-1333 CL9 Unbuffered
Hard Drive
2 x OCZ Vertex 3 120 GB in RAID 0
Graphics
On-board graphics from each platform
Power Supply
Included power supplies from each platform
System Software And Drivers
Operating System
Windows 7 Ultimate 64-bit
Ubuntu Linux 10.10 64-bit


An abbreviated test suite was used for this round-up, since the performance differences between platforms ended up being very slight (in light of common processors and memory). All tests were run using a clean installation of Microsoft Windows 7 64-bit, except the Folding@Home test that used Ubuntu Linux 10.10 64-bit. The older version of Ubuntu is probably the most widely-used distribution for this application, as it produces consistently high performance.

Benchmarks and Settings
Applications
Adobe After Effects
Version: CS5.5
Tom's Hardware Workload, SD project with three picture-in-picture frames, source video at 720p, Render Multiple Frames Simultaneously
Adobe Photoshop
Version: CS5
Tom's Hardware Workload, Radial Blur, Shape Blur, Median, Polar Coordinates filters
Adobe Premiere Pro
Version: CS5.5
Paladin Workload, Maximum Render Quality, H.264 Blu-ray profile
Euler3D
CFD simulation over NACA 445.6 aeroelastic test wing at Mach .5
3ds Max 2012
Version: 10 x64
Rendering Space Flyby Mentalray (SPECapc_3dsmax9), Frame: 248, Resolution: 1440 x 1080
Visual Studio 2010
Compile Chrome project (1/31/2012) with devenv.com /build Release
Folding@Home
6903 Work Unit under Ubuntu 10.10 64-bit
-smp -bigadv CLI client parameters
Synthetic Benchmarks and Settings
Cinebench 11.5
CPU Test, Built-in benchmark
3. Supermicro 6027R-N3RF4+: Layout And Overview

The Supermicro 6027R-N3RF4+ is a complete barebones kit composed of an X9DRW-3LN4F+ motherboard and CSE-829BTQ-R920WB chassis. Both parts are available separately, but Supermicro sells the combination with cabling already run, routed, and secured to optimize airflow. This does save quite a bit of time compared to purchasing each piece and assembling. Supermicro also offers Intel CPUs and other components on its price list, so it is possible to not only purchase the company's components, but also to configure a fully-assembled system.

Let's take a look at the 6027R-N3RF4+ barebones package.

Looking at the front of the CSE-829BTQ-R920WB chassis, ten 3.5" disk trays are perhaps most prominent. Generally, this provides room to install two boot drives and a fully-populated eight-channel SAS controller. Three-and-a-half-inch disks are generally used as mass storage, with 10 000 and 15 000 RPM spindles becoming less common (in favor of 2.5" SAS drives or solid state drives when performance is the priority), making 3.5" the form factor of choice for larger, slower 7200 RPM SAS or SATA devices.

You can also see Supermicro's LCD status panel, along with USB ports, status LEDs, and buttons. The company makes room for a slim optical DVD drive near the top of the chassis, which is less common these days due to the remote image mounting capabilities of many management interfaces.

A quick look at the drive tray makes it pretty obvious that it'd be difficult to install a 2.5" device without utilizing an adapter. Supermicro does sell adapter hardware, but given the dominance of 2.5" storage for high-IOPS workloads, I do hope Supermicro's next design incorporates the ability to use 2.5" drives in 3.5" bays through either a unified drive sled or adapter.

We clearly see that the CSE-829BTQ-R920WB is built to accommodate custom form factor motherboards (WIO+). Proprietary form factors are very common when vendors design systems with expansion slots using riser cards, which the 6027R-N3RF4+ does utilize. This is important because the number and location of expansion slots via risers is not standardized, as you might expect on the desktop. When you purchase a system based on a proprietary form factor from a vendor like Supermicro, Dell, HP, or IBM, the ability to upgrade later on is often limited as a result.

There are four hot-swap 100 CFM 80 mm fans that dissipate heat from not just the motherboard, but also the passive CPU heat sinks through the use of clever ducting with an air shroud. Hot-swap fans in a chassis of this price are standard because, if a fan fails, you need the flexibility to pull it out and install a new one from the top of the chassis without touching the motherboard. This is doubly important in 1U and 2U enclosures, where CPUs are often passively cooled.

Moving to the rear of the chassis, the Supermicro 6027R-N3RF4+ includes redundant hot-swap 1U 920 W power supplies with 80 PLUS Platinum ratings, configured one on top of the other. These PSUs have a replacement part number of PWS-920P-SQ, with the "SQ" designating Supermicro products marketed as "Super Quiet". We find that interesting in a 2U server chassis destined for data centers. The 80 PLUS Platinum rating means that these PSUs can achieve up to 94% efficiency, an important metric in racks, where power delivery and cooling are finite. PSU efficiency plays an important role in lowering overall infrastructure load.

In order to remove one of the platform's power supplies, push the latch securing it and pull the unit's handle. Inserting a power supply is as simple as pushing it into a vacant slot.

4. Supermicro 6027R-N3RF4+: Layout And Overview, Continued

You'll also find four full-height expansion slots around back, along with three half-height slots. As we're about to see, only two of the three half-height bays can be used at a time. Otherwise, the motherboard's memory interferes with the third.

Supermicro's chassis employs fixed cut-outs for the motherboards it accepts, rather than a larger block for a customizable I/O shield. We're looking at a proprietary design, so it's unlikely that you'd ever put another motherboard in this chassis. So, the lack of upgradability in this application is both acceptable and fairly standard. Much like servers designed by Dell, HP, Sun, IBM, and Fujitsu, engineers build these enclosures to optimize expandability and airflow, not to adhere to standard motherboard form factors.

A quick peek at the available ports reveals a legacy PS/2 keyboard and mouse combo port, serial connectivity, VGA output, and four USB ports. There is a total of five 8P8C jacks: one above a pair of USB ports, and four in a row to the right. The one above the USB ports originates from a Realtek RTL8201N, and is used for IPMI 2.0 remote management functions. Some server vendors use a shared port for remote management, but I personally prefer a dedicated port, since I set up LANs to VPN into for management. Installing new servers in environments like this is very easy because you know that the cable from the management NIC goes to the management network. The other four ports are serviced by an on-board Intel I350-AM4 controller, which is becoming standard on the newest-generation servers.

As mentioned, the Supermicro X9DRW-3LN4F+ supports up to six add-in cards. The sextet is PCI Express 3.0-compliant, and exposed through risers. Two are full-height, full-length x16 slots, one is a full-height, full-length x8 slot, one is a full-height, half-length x8 slot, and two are low-profile slots.  

Supermicro utilizes Intel's C606 PCH in its design, which gives the platform two SATA 6Gb/s ports, two SATA 3Gb/s ports, and eight 3 Gb/s SAS ports directly from the PCH. This is a big departure from previous generations, where an engineer would typically dedicate PCIe connectivity to an on-board SAS host bus adapter or RAID controller.

Intel's C606 chipset's SAS ports are really targeted at the 3.5" high-capacity storage market, where 3 Gb/s is generally enough throughput. The company is not being as aggressive as originally thought (after all, LSI is one of Intel's partners), as it was originally believed that the SCU would feature 6 Gb/s SAS connectivity. For this generation, at least, all three participating vendors do offer LSI-based controllers as upgrade options. We just aren't seeing them, since we requested that manufacturers leave them off.

One thing you do see on the board, directly resulting from this integrated chassis/motherboard package, is that one of the two SAS-based SFF-8087 connectors is angled to provide easier cable management for the four-link cables.

Each CPU and its associated memory is placed in-line, but slightly offset. That's a standard configuration for dual-socket server motherboards. Supermicro manages to fit 12 DDR3 DIMM slots per CPU, yielding a maximum memory configuration of 192 GB using unbuffered ECC DDR3 memory or 768 GB of registered ECC DDR3 memory.

Those 12 slots mean that Supermicro's enclosure can only hold two cards across its three low-profile expansion slots. They also mean that Supermicro's cooling solution is not compatible with the standard 91.5 x 91.5 mm LGA 2011 footprint. Instead, it uses a narrower mounting orientation (look for "S" at the end of Supermicro SKUs to ensure you have the slim version.) To address its unique coolers, Supermicro provides two SNK-P0048PSC (note the "S" in PSC for slim LGA 2011) passive 2U heat sinks for our 135 W Xeons.

5. Supermicro 6027R-N3RF4+: Management Features And Serviceability

Supermicro utilizes the AMI MegaRAC line of management processors to provide out-of-band remote management. This is not a complete review of the management features and BIOS settings (that'd take 10 pages per system). But I did want to provide a quick overview for anyone not yet familiar with these capabilities.

One of Supermicro's biggest differentiators is the ability to mount images from network shares. Although that doesn't seem like critical functionality, it is very useful. A common practice is to store drive images on a network share. Using most other solutions, you have to map a network share to the drive, then use remote management to mount those images to the server. The issue there is that if you're installing an operating system, for example, and then lose network connectivity on the system you're using (let's say a notebook goes to sleep when you walk away), the image is no longer accessible and the installation fails. Also, bandwidth is consumed from the administration end point because data has to be passed from the network share to the administrator's machine, and then from the administrator's machine to the server. Directly mounting images on the server takes your admin system out of the equation.

You can see that there are fairly standard MegaRAC power controls in Supermicro's implementation.

The company provides a lot of settings related to user authentication and e-mail alert notifications.

Monitoring is another necessity in remote management, and Supermicro provides access to fan speeds and temperatures. Personally, I use Supermicro IPMIview 2.0 rather than the IPMI management most of the time because it lets me monitor more than one system at a time.

Using the KVM-over-IP features, you can do a few things that SSH, VNC, and RDC cannot do, such as remote troubleshooting from POST. An administrator can use the Java-based application to see exactly what is being shown on the VGA port, and then use his keyboard and mouse as if it were plugged into the server's own USB ports (with a few exceptions). For commands like Ctrl-Alt-Del that normally trigger a local system event, MegaRAC uses macros to get those commands out remotely.

Supermicro's solution exposes the 3 Gb/s SAS ports on Intel's C606 chipset, and their status is easily monitored in the BIOS.

As in the automotive world, green is a key concept in data centers since there are real cost-savings to be had. Supermicro does have a number of power-saving features that are configurable in its firmware.

One more feature before we move on to Tyan's offering: Supermicro offers the ability to control iSCSI settings in its BIOS. This is fairly important if you boot servers to iSCSI shares.

6. Tyan GN70-K7053: Layout And Overview

Tyan's approach to the server space is traditionally different than Supermicro's. It used to sell its motherboards on their own. Recently, though, the company started seeing demand for enclosures designed specifically for its motherboards, and it now has a chassis line-up, too. Tyan sent along its S7053 motherboard and KGN70M1 2U chassis in a barebones package called the GN70-K7053.

The front of the Tyan KGN70M1 chassis presents few surprises. There are eight 3.5" hot-swap drive bays that feel solidly-built. The top-right  bay is used to house an optical drive, and you'll find a standard assortment of USB ports and LED indicator lights.

It's pretty apparent, given the copious ventilation up front, that Tyan is looking to keep lots of air moving through its case. And if you're wondering, the eight-bay configuration is also fairly common, since most storage controllers feature port counts in multiples of four. And once you go over eight, you're typically looking at an add-in SAS expander on a RAID card or HBA.

Tyan's drive assembly warrants attention, too. The company provides an insert in its 3.5" tray that lets you install a 2.5" disk or SSD in the hot-swap bays.

Tyan employs a fairly standard 2U layout with the ability to not only support ATX and EATX form factors, but also EEB- and CEB-based motherboards as well.

The middle of the chassis accommodates two rows of four hot-swap fans. The dual-row design gives Tyan the flexibility to deliver redundant cooling in the event a fan fails during operation. As you can see, each fan has a small carrier that enables simple hot-pluggable installation. A failed fan causes the chassis to trigger an alarm, and a small LED indicator goes red when a fan fails or is removed. Since each fan uses just over 16 W at full speed, Tyan equipped our sample with a single row of fans.

To the right of the mid-plane fans, Tyan uses a special distribution board. Power from the redundant PSUs is routed through the board's PCB and off to fans, drives, and the motherboard itself.

As mentioned, the KGN70M1 includes dual redundant power supplies in a side-by-side configuration. The two power supplies are 770 W 80 PLUS Gold-rated units that can be pulled out by releasing the orange lever and pulling on the black handle. Tyan does employ ducting, and it uses the space above the power supplies for chassis exhaust.

7. Tyan GN70-K7053: Layout And Overview, Continued

Because the KGN70M1 is not a chassis designed with risers in mind, instead utilizing half-height expansion cards, you can see that the rear panel has enough room for six shorter peripheral devices. Often, chassis employing side-by-side power supplies leave room for this exact sort of expansion scheme. Again, this is a fairly standard setup.

Unfortunately, our review unit was missing its metal I/O shield, and we didn't have a spare to match Tyan's S7053 motherboard. So, that space is uncovered.

The rear-panel reveals that the S7053 includes legacy VGA and serial ports in case you need to attach a serial device or monitor into the server. Although KVM-over-IP is fairly widely accepted at this point, the possibility exists that you'll need to make a physical connection at some point instead. Next to those ports, you find two stacks that include a pair of USB ports and an 8P8C connector attached to an Intel gigabit Ethernet controller. Another pair of gigabit-capable connectors sits on the far-right side of the motherboard's integrated I/O.

Tyan really goes the extra mile here in providing four USB ports, rather than the more common pair. If you're setting up a server using a VGA monitor, mouse, and keyboard, that typically leaves little additional connectivity for a driver-laden USB thumb drive, for example.

All four 8P8C female jacks are driven by an on-board Intel I350-AM4 network controller, which is one of the company's newer-generation chipsets.

The board itself features six PCIe 3.0-capable expansion slots, four of which are eight-lanes wide, and two of which offer 16 lanes of connectivity. Tyan uses open-ended x8 connectors, which we wish that more motherboard vendors would do as well. The benefit of an open-ended slot is that they accept x16 devices as well at half-bandwidth, so long as they clear other components along their path.

You can see near the edge of the board that Tyan is using Intel's Patsburg-A (C602) PCH. So, you find two SATA 6Gb/s ports, along with four SATA 3Gb/s connectors derived from the Intel core logic. Tyan also includes an LSI SAS 2308 RAID controller with support for standard RAID levels without parity, which we'd expect from a controller with no associated battery or NAND-backed memory. Because our review guidelines specifically requested vendors not to send motherboards with on-board third-party storage controllers, we won't go into detail on this feature except to mention that the kit comes wired to utilize LSI's controller with the KGN70M1 2U chassis.

Both CPU interfaces are set up for the standard LGA 2011 mounting pattern, which is similar to the LGA 1366 layout. Each processor and its associated memory is placed in-line, but slightly offset (again, standard for dual-socket server motherboards). There are eight DDR3 DIMM slots per CPU, yielding a maximum memory configuration of 128 GB using unbuffered ECC DDR3 memory, 256 GB of registered ECC DDR3 memory, or 512 GB using LR-DIMMs (load-reduced DIMMs).

8. Tyan GN70-K7053: Management Features And Serviceability

Like Supermicro, Tyan utilizes AMI's MegaRAC management processor to provide IPMI 2.0 and KVM-over-IP functionality. Tyan uses a less customized version of the MegaRAC than Intel and Supermicro, so you clearly see the MegaRAC branding. Nevertheless, much of this is going to look a lot like what we already saw from Supermicro's MegaRAC implementation (along with Intel's, as we'll soon see).

Here, Tyan provides standard remote mounting features. The company unfortunately does not offer the ability to mount a remote network share.

Tyan also exposes the standard power controls found in the MegaRAC management solution. Like Supermicro, Tyan provides a lot of settings related to user authentication and e-mail alert notifications. These settings get used to a varying degree, depending on the environment.

Another key piece of the solution is monitoring server health, including voltages, fan speeds, system events, and alerts.

Using the KVM-over-IP features, you can do a few things that SSH, VNC, and RDC cannot do, such as remote troubleshooting from POST. An administrator can use the Java-based application to see exactly what is being shown on the VGA port, and then use his keyboard and mouse as if it were plugged into the server's own USB ports (with a few exceptions). For commands like Ctrl-Alt-Del that normally trigger a local system event, MegaRAC uses macros to get those commands out remotely

Tyan offers similar BIOS ECC memory mode settings as Supermicro and Intel. The options are certainly more advanced than what you'd find on a Xeon E3-series board, since you can use registered DIMMs on the E5-based platform.

Tyan does not use the more expensive C606 chipset implementation with eight ports of SAS connectivity, which is understandable since the S7503 includes an LSI SAS controller.

Power management features are enabled though Tyan's BIOS, a fairly standard practice on server motherboards.

Before we move on to Intel's submission, we see that Tyan offers BIOS-based fan control. Server boards tend to use four-pin PWM-controlled fans, which are able to ramp fan speed up and down to maintain cooling without wasting power.

9. Intel R2208GZ4GC: Layout And Overview

Third-up is Intel's R2208GZ4GC "Grizzly Pass" kit, which the company touts as being highly customizable. To that end, we see that the front of the chassis is partitioned off into three sections to facilitate different drive options. The kit came with both 10 GbE and LSI RAID controller options, which I removed for testing and price comparisons. Our test mule has eight 2.5" hot-swappable bays, but the chassis does allow for a second block of hot-swappable bays if you want to configure it for 16.

Like Supermicro and Tyan, Intel makes room for a slim optical drive. Most 2U enclosures designed with high density in mind sacrifice this space (along with front-panel I/O) to cram as many as 24 drives into the front of the chassis. All three of the samples sent to us include two front-panel USB ports and a slim optical bay, though. Whereas the Supermicro chassis also features a front-accessible DE-9 serial connector, Intel's solution includes an HD-15 VGA connector. Intel's front-panel connections are tailored for KVM.

It's pretty apparent that Intel is using a heavily customized implementation, and its PCB is absolutely huge. Officially, the S2600GZ4 motherboard is a proprietary 16.5 x 16.5" form factor. Just to give you an idea of this board's size, a typical quad-socket AMD G34-based platform with 32 DIMM slots fits in a 16.5" x 13" form factor. The benefit of such a large PCB is that you can see Intel's enclosure is built to exploit the additional size.

There are five 80 mm fans in the middle of the chassis that blow air through thermally-sensitive components. A clear plastic duct guides air through passive CPU heat sinks. Whereas Supermicro's shroud channeled air all the way through to the back of its enclosure, Intel employs a shorter duct, since its chassis uses much of the space around back for PCIe rises and power supplies.

Two 80 PLUS Platinum-rated 750 W power supplies connect directly to the motherboard. This is a significant difference between Intel's implementation and the competition from Tyan and Supermicro, which utilize an intermediate distribution board for power. 

Both redundant power supplies are removed by pushing on the teal lever and pulling the handle back. Intel uses a side-by-side configuration and does exhaust air from above the PSUs.

10. Intel R2208GZ4GC: Layout And Overview, Continued

Expansion slots are enabled through two riser cards, which allow for up to six single-slot add-in devices in total. Intel also offers its own proprietary network and RAID cards. You can see the spot at the bottom-right corner of the rear panel that takes NICs, such as a dual-SFP+ X540-based Intel 10 GbE upgrade. Intel armed our machine with a dual-port 10 GbE controller, but we pulled it out to maintain better parity with the other systems, per our round-up request.

The rear I/O panel does not use a standard shield design, similar to Supermicro's offering. Intel does this to accommodate a pair of what it calls PCIe 3.0 super-slots (24-lane slots that enable three x8 connectors each) for expansion. Built-in I/O includes four 8P8C female jacks driven by Intel's bridgeless i350 controller, three USB ports, a serial-over-LAN port, VGA output, and a dedicated KVM-over-IP port enabled by a small daughter card.

Like Supermicro, Intel uses a shroud to guide airflow around its CPUs and memory. Intel's cooler employs a much lower-profile design, though, that looks like it'd even work in a 1U enclosure. As we'll see in the benchmarks, both solutions are able to keep our Xeon E5 processors at their full multi-core Turbo Boost clock rates for our entire test period. Atop of the Lexan air shroud, Intel provides two 2.5" mounting points probably best used for SSDs.

As we've mentioned, Intel sells a variety of proprietary add-on board options that let you install an upgrade without plugging up a PCI Express slot. Some of the options include 10 GbE Ethernet, LSI-based RAID controllers, and specialized management cards. In the picture above, you can see both the dedicated KVM-over-IP board and LSI-based storage card. Neither Tyan nor Supermicro make KVM-over-IP something you have to buy separately, and with so many servers shipping with that functionality already, it would have been nice to see Intel make that a standard feature, rather than an upsell.

Intel's server system, like Supermicro's, exposes eight 3 Gb/s SAS ports via a pair of SFF-8087 connectors in the middle of the board. Given that this platform has eight 2.5" bays, 6 Gb/s connectivity would have made more sense, which is probably why Intel outfitted our review unit with the LSI SAS 2208-based RAID card.

11. Intel R2208GZ4GC: Management Features And Serviceability

Intel's server management solution is also based on the MegaRAC product, so many of these features will look fairly familiar from the previous two machines. Intel does give the interface some attention, though.

One great feature is a check box to boot to the firmware. If you've used MegaRAC in the past, then you know that the Java-based KVM does have a bit of latency associated with it. If you need to get into a remote motherboard's BIOS, checking that box makes that operation a lot easier than trying to time an F2 or Del keystroke.

Intel also gives you access to power statistics, shedding some light on minimum, current, maximum, and average consumption. What you see in the shot above only reflects our power use during setup, not during the benchmarks. We'll give you those numbers shortly.

Another feature called Node Manager that facilitates management of each machine in the context of a complete rack. You can set a power budget that limits consumption, allowing a number of systems to operate concurrently without exceeding a data center's limits.

One notable feature that the Intel Management WebGUI does not include is the ability to reset the baseboard management controller (BMC) from the Web interface. Though this doesn't sound like a show-stopper to anyone without experience using MegaRAC products, it's a problem if you get a Video Socket Error, where the Java KVM cannot open and refresh video. In a remote data center, if the BMC errors out, you could end up needing hands on-site to perform a cold reboot. Intel should be able to address this, and it is something that every MegaRAC-based solution needs. I encountered this exact situation during a reboot for a BIOS setting change with the Intel platform, and I see this error from time to time during my normal work on ServeTheHome with boards from various vendors using the MegaRAC solution. And, if you search online, you'll see the problem isn't uncommon. The easiest way to fix it is rebooting the BMC.

Intel also offers a neat feature called the Virtual Front Panel, which shows the enclosure's front LEDs. It is an interesting feature that comes in useful if you want simple diagnostic information from a remote location.

With that said, Intel does provide much more complete sensor information in the IPMI interface.

Intel's KVM-over-IP feature, seen in the shot above, is similar to the implementations from Tyan and Supermicro.

12. Pricing, Warranty, And Support Comparison

I recently read that the average selling price for a Supermicro system was around $1700. These barebones systems are all in the ballpark of that figure, and hitting a typical price range was a major consideration when framing this round-up. We've already covered the hardware and software side of each submission, so let's take a look at how pricing factors in.

Pricing Comparison
Supermicro 6027R-N3RF4+
$1515
Tyan GN70-K7053 (S7053 + KGN70M1)
$1700
Intel "Grizzly Pass" R2208GZ4GC$1900


Of course, you can add additional components to affect that price (such as Intel's management board). Intel, Supermicro, and Tyan all have options to customize their platforms, so your take-home cost is going to vary. These systems to represent a good basis for comparison, though. Price-wise, Supermicro and Tyan both managed to duck in under our expected $1800 price point, with Supermicro coming in significantly below that price target. Intel's system turned out to sell for a bit more, which makes sense given its highly-customized platform.

Service, warranty coverage, and support are also important points to consider, particularly when it comes to buying servers. When you purchase a machine from HP, Dell, IBM, Oracle/ Sun, or Fujitsu, you can specify anything from mail-in service to 24x7 support with a guaranteed on-site response time (for instance, 24x7 four-hour service). In the world of barebones servers, though, you'll typically have a reseller to assume the role of support provider. So, if you buy from a VAR that handles integration of the parts you pick, that company also arranges service. Your interactions are through the integrator, rather than the company manufacturing the system's motherboard or chassis.

Here's a quick reference of warranty coverage:

Warranty Comparison
Value-Added Reseller Warranties May Differ
Supermicro 6027R-N3RF4+
One-year warranty from Supermicro, 8x5 standard support hours (upgraded coverage available), advance RMA available, pre-paid service available, extended warranty available from Supermicro
Tyan GN70-K7053 (S7053 + KGN70M1)
Three-year warranty from Tyan/ MiTAC, 8x5 standard support hours (upgraded coverage available)
Intel R2208GZ4GC
Three-year limited warranty, extended warranty available from Intel


Update, 12/8/2012: It was recently brought to our attention that Supermicro now sells it standard complete systems with three-year parts and labor warranty coverage, along with one year of cross-shipment and 24-hour tech support.

Typically, start-up VARs and systems integrators are the interface point to the OEM when something goes awry. Once these partners become large enough, they typically receive a larger discount, allowing them to purchase spare parts instead of having to get an RMA from the manufacturer when a component goes out. So, you'd generally get your warranty work from a VAR, and not the manufacturer.

If you're the one buying the components and building servers, either as an SMB or end-user, this becomes a major consideration, since there is no reseller on which to fall back. Advanced RMA becomes especially useful, since it typically cuts the time it takes to get a replacement by days. This service is becoming more popular on desktop motherboards, and Western Digital has offered it for years.

Both the Supermicro and Tyan platforms can be beefed up with additional functionality to hit Intel's price point. However, both companies submitted their products for the round-up in accordance with what was requested.

13. Benchmark Results: Adobe CS 5, 3ds Max, And Cinebench

Each of our three samples employs Intel's Patsburg chipset with two Intel Xeon E5-2690 CPUs and the same memory. Overclocking is not supported, aside from the dynamic frequency adjustments attributable to Turbo Boost technology. What we're looking for, then, are dips in performance that could be caused by throttling as a result of getting too hot, or Turbo Boost not spinning up as high on one platform as the others. All of these systems are designed to run at full speed 24x7.

The trio is tied in Adobe's Premiere Pro.

After Effects similarly finishes very close. Whereas Premiere is a well-threaded metric, this test tends to not favor clock rate more. The fact that the results come close again suggests we're seeing similar impact from Turbo Boost.

Adobe Photoshop CS 5 is still very popular. But with Photoshop CS 6's recent launch, Adobe did introduce certain performance enhancements. Still, we can see that identical processors and memory yield very similar performance.

3ds Max 2012 is a popular 3D modeling program used by many professionals. Once again, we see no evidence of thermal throttling.

Cinebench 11.5 demonstrates impressive performance, more than tripling what the just-launched Xeon E3-1200 V2 series can do. Again, though, there is very little differentiation between platforms.

14. Benchmark Results: Compiling, Folding, And Euler

We've seen a pattern emerge that will continue: the three barebones servers offer very similar performance. Again, this is what we would have expected. The big question is whether one will throttle due to thermals as we stress a pair of 135 W CPUs.

Euler3D has become a fairly popular scientific benchmark for CFD modeling. Each platform again performs very similarly.

Compiling Chrome using Visual Studio shows a similar performance story.

Using Folding@Home, we were able to heat the CPUs at 100% load for hours. These measurements were taken after six hours of running continuously on a single work unit. As we can see, the averages are more or less identical, indicating not throttling during our run.

From a performance perspective, these servers are very much similar, which is what we expected (and indeed wanted) to see given similar CPUs, memory, and core logic. This means that there were no major thermal issues to report.

15. Power Consumption And Noise Comparison

We used Ubuntu Server 10.10 and Folidng@Home for power consumption and acoustic testing. It's fairly common to see these measurements taken fairly quickly after a configuration is presented with  load. But instead of taking a reading 15 minutes into a workload, we let these servers run at 100% utilization for six hours each. This actually yields an appreciable difference in power consumption and noise compared to the shorter testing used on desktop hardware, and is probably closer to what a server might see in a data center. We picked Folding@Home because it is a real-world GROMACS-based application used for research purposes, and a big work unit takes 24 hours or more for the machine to complete. It also scales fairly well from four to 64 cores, making it useful for generating high CPU load over long periods. Both Linux and Windows clients are available, but the Linux client tends to be a quite a bit faster on the same hardware, and Linux gives options for better thread management.

Power consumption and noise testing was performed using each motherboard's PWM-controlled fan settings. This is worth noting because there is one school of thought that suggests you should keep fans at full speed in order to push as much air through the chassis as possible. A side benefit of pegging fans at full speed is that there is no ramp up or down, putting additional wear on the fans. The other school of thought, and one that I ascribe to, is that motherboard PWM fan control is good enough to keep components cool while saving power. If workloads are relatively even, there is not much ramp up or down anyway, allowing the coolers to spin at lower average speed. Just to give you an idea, at full speed, the fans used in these barebones units can draw between 15 and 18 W each. With four to eight fans per system, that's not a negligible amount of power. Data centers typically have restrictions on power delivery, so cutting back may help save overage charges.

An Extech 380803 True RMS power meter was used to measure power consumption of each system. Measurements were taken at the six-hour mark and averaged over three runs. The systems were allowed to cool for three hours between runs in a fully powered-off state.

Supermicro and Intel achieve almost identical power consumption. Tyan isn't far behind. However, there is a measurable delta between the three systems. There is a distinct possibility that the difference is due to Supermicro and Intel using 80 PLUS Platinum-rated PSUs, while Tyan arms its machine with less-efficient Gold-rated power supplies. Also, Tyan uses a separate PCB to distribute power around the chassis.

For sound testing, we used an Extech 407764 NIST-calibrated digital sound meter with a 15-foot microphone extension cable so that we could monitor from another room to preserve a 32 dB(A) sound floor (the Extech 407764 is rated for 30-130 dB sensitivity.) The idle measurements were taken 30 minutes after the system booted into Ubuntu Server, and the load readings were taken before each Folding@Home six-hour run was completed. Three runs were averaged for the scores below.

Intel does well here, as does Supermicro. The Tyan chassis is a bit louder (particularly at idle). But being a 2U enclosure, there is significantly less noise than if these were 1U machines. All three servers are too noisy to put in the middle of an open office or in a home. However, in a data center, these noise levels are expected.

Realistically, these machines will run under at least some load most of the time. So, while idle noise is recorded for the sake of completeness, the results are less relevant in the real world than the load numbers.

16. Whose 2U Server System For Xeon E5 Is Best?

Despite the fact that we're looking at a trio of 2U server barebones systems capable of taking two LGA 2011-based processors, Intel, Supermicro, and Tyan all take very different approaches in getting there.

Tyan offers a 3.5" storage-oriented chassis (with 2.5" adapters) and a very standard motherboard design. Supermicro's system is more customized, including a PCIe riser, 80 PLUS Platinum PSUs, and the highest number of hot-swap bays of any enclosure in our round-up. Intel sells an even more customized riser design, similarly high-efficiency PSUs, configurable storage, and the largest price tag.

Each solution can be improved upon, we think. Tyan could benefit from more efficient power supplies to match Supermicro and Intel. Moreover, its adoption of a standards-compliant motherboard limits it to low-profile expansion cards. That's good if you want to upgrade the motherboard down the road, but that's a fairly rare occurrence in the server world, and it limits what you can plug in today. Supermicro offers fewer full-length slots than Intel, and Intel takes the advantage for custom expansion that takes networking, storage, and management cards without plugging up the PCIe slots. Unfortunately, Intel also requires that you buy a daughter card for dedicated IPMI 2.0/KVM-over-IP support, and it didn't give us a BMC reset through the MegaRAC GUI. Limited to eight 2.5" drives in the trim we received it, it also costs the most.

With those few critiques mentioned, each product has its strong points as well. Naturally, Tyan's solution is the easiest to upgrade in the future, given its standardized form factor. We also like the hot-swap bays with 2.5" adapters, able to accept either drive size, and the fact that Tyan also includes an on-board LSI-based SAS controller. For that, Tyan's system drops into the middle spot for pricing. Intel uses an absolutely massive motherboard to provide a very well-integrated solution with arguably the most expandability. The company's customization of the IPMI 2.0 WebGUI is attractive and easy to navigate. Meanwhile, Supermicro differentiates its solution with value pricing, a lot of room to add additional components, and high-capacity, efficient power supplies. In addition, Supermicro includes the largest number of hot-swap storage bays, on top of a nicely customized MegaRAC interface.

After extensive time spent testing and replacing components in each of these 2U servers deployed to a rack, the picture above sums up the order in which I'd rank them. The Supermicro chassis blends expandability and a feature-complete specification sheet into a package that compares well to the competition at a much lower price point (and with additional on-board options available.) Intel's submission actually came in above our round-up's price range. However, the inclusion of storage and networking expansion modules also gives the R2208GZ4GC a notable advantage in configurability, while preserving all six PCIe slots. Tyan's solution is very conservative, which pays off if you're a builder who plans to re-use the chassis for another motherboard in the future. We think it has the best redundant cooling implementation of the three contenders, too. The only reason it finishes behind Supermicro and Intel is because the other two vendors give you more room for memory, add-in expansion, and higher-efficiency power supplies.

With all of that said, even though these server systems are similar in many ways, there is clearly a niche for each. Realistically, an SMB buying from a VAR is going to end up with whichever vendor that reseller is using, and the configuration that makes the most sense for a given application or usage model.

Of course, we welcome any of your thoughts or feedback as look to continue our coverage of more enterprise-class products and technologies.