Taiwanese authorities have been stepping up their investigation into the alleged AI GPU smuggling by some Supermicro employees, and it was recently reported that the police “raided” the company’s Taipei office. However, Digitimes reports that the firm is pushing back against this characterization, with Supermicro insisting that it’s cooperating with the investigation. Instead, it voluntarily gave investigators access to the workstations and electronic devices of the employees suspected of violating U.S. export controls while also placing them on administrative leave.

“Supermicro’s offices in Taiwan were not raided by any government authorities,” Supermicro Chief Revenue Officer Matt Thauberger told customers and partners in a written statement. He also confirmed with the Taiwanese government that the company was not the target of any investigation and that it has been cooperating since May of this year. "We have zero tolerance for anyone who violates the law or our internal policies," he also wrote in the letter.

Supermicro insists that it is cooperating with officials in the investigation and even gave them access, and as such the police intervention wasn't technically a “raid.” It remains unclear, though, how long the company knew about the pending police action before the authorities arrived on the premises.

This investigation is part of Taiwan’s push to investigate the alleged smuggling of Nvidia AI chips into China through its territory. The island does not have any laws that echo the U.S.’s export controls, so it’s using a loose interpretation of other regulations. This is similar to what Singapore is doing, which recently seized a $42 million mansion and $772k stashed in a bank account that are owned by alleged AI GPU smugglers. But because it also does not have export control laws like Taiwan, the accused are instead charged with fraud and money laundering.

Aside from the case brought against Supermicro employees in Taiwan, three other individuals linked to the company, including its co-founder, Yih-Shyan “Wally” Liaw, have been arrested in the U.S. on charges of conspiring to violate the Export Controls Reform Act. They allegedly used a hairdryer to soften the glue on thousands of serial numbers on the banned servers and moved them to dummy units to make them harder to track. They were then reportedly shipped via a Thailand-based government-related entity before landing in Chinese tech giant Alibaba’s warehouses.

Although Supermicro isn’t directly accused in these two cases, the fact that many of its employees are being investigated and charged is probably raising concern among its partners and customers. It has even gotten to the point that Nvidia CEO Jensen Huang has urged it to fix its export compliance controls. This is likely the reason why the company has taken moves to clarify the situation, which has already caused its stock price to slide by 8% in U.S. trading.

Supermicro is showing off its upcoming machines based on AMD’s 6^th Generation EPYC ‘Venice’ processors, Instinct MI450 accelerators, and, of course, Nvidia’s Vera Rubin-based solutions. Arguably, the most important product that the company demonstrated at its CEO’s keynote is the VR200 NVL72 rack that uses Vera CPUs and Rubin GPUs. The machine, just like other upcoming liquid-cooled systems from Supermicro will use the company’s all-new coolant that has a 1000x higher electrical impedance compared to today’s mixtures, which may be quite important for next-generation AI machines.

“Our new coolant [uses a new] formula [and] reaches up to 1,000 times higher electrical impedance than a standard cooler,” said Charles Liang, chief executive of Supermicro, during his keynote speech at Computex. “In case there are small leaks, when you have a high electrical impedance, the system will not [shut down] [and will] keep running.”

Conventional water-based coolants used in direct liquid cooling systems have electrical conductivity (albeit lower than water), so if coolant leaks onto a motherboard, GPU, power delivery circuitry, or connectors, it can create leakage currents or even short circuits. A coolant with 1000 times higher electrical impedance is far more resistant to current flow, which reduces the likelihood that a minor leak will immediately shut down a system or damage electronic components. This is important as modern rack-scale AI solutions like Nvidia’s VR200 NVL72 are rumored to cost around $8 million, so their protection is crucial. Also, reducing downtime in AI data centers is important as these machines must make money for their owners.

Unfortunately, Supermicro’s claim is difficult to evaluate because the company does not disclose specifications of the coolant, such as conductivity (µS/cm), resistivity (MΩ·cm), or dielectric strength (kV/mm). It also remains tight-lipped about the baseline coolant used for comparison. Since modern water-glycol coolants are already fairly resistant, a 1000-fold improvement sounds significant, but without the actual details, it is impossible to determine the practical magnitude of the advancement. What Supermicro claims is that a minor coolant leak would be less likely to force an immediate server shutdown, which means lower downtime risk in large AI deployments. However, we have no idea whether the new coolant behaves more like a dielectric fluid and less like water-glycol coolants.

The new coolant will be used in all new liquid-cooled systems from Supermicro, including the upcoming AMD Helios and Nvidia VR200 NVL72 machines. Given that Nvidia nowadays does not leave many ways for its partners to differentiate, the coolant with 1000 times higher electrical impedance than conventional coolants will likely be among the key selling points of Supermicro’s Nvidia-based offerings.

Speaking of Nvidia’s AI systems, Supermicro is on track to release its Vera Rubin-based systems in the second half of this year, just like other makers. Supermicro intends to offer both NVL72-rack-scale machines based on Nvidia’s Vera CPUs and Rubin GPUs, as well as Rubin DGX systems with different processors.

As for AMD’s side of matters, Supermicro is also on track to release MI455X-based Helios rack-scale solutions in the second half of the year, though the company does not disclose whether these use UALink or UALink over Ethernet interconnections. In addition, Supermicro is also on track with 1-way and 2-way servers based on AMD’s 6^th Generation EPYC ‘Venice’ processors that are made using TSMC’s N2 process technology. These machines are also expected to hit the market in 2026.

A day after Taiwanese authorities seized 50 Supermicro servers and arrested three people on suspicion of smuggling them into China, Supermicro has clarified that it "worked closely" with law enforcement in Taiwan leading up to the arrests and that it will continue to do so.

This comes days after Nvidia CEO Jensen Huang called on Supermicro to "improve regulation compliance," and months after Supermicro co-founder Yih-Shyan "Wally" Liaw was charged by U.S. prosecutors for allegedly smuggling $2.5 billion worth of servers into China.

Here is the statement in full:

"Supermicro is committed to protecting our advanced technologies and intellectual property, and we are proud to have worked closely with Taiwanese authorities on the recent event, helping to prevent the illicit diversion of our highly sought-after systems into the restricted China market. Our collaboration with authorities in Taiwan resulted in the arrest of three suspects and the seizure of 50 servers that had been deceptively acquired after being sold by Supermicro to an authorized reseller. We thank local law enforcement and legal officials for their vigilance and professionalism.

The initial transaction – Supermicro's sale of products to an authorized reseller – followed a rigorous vetting and review process that exceeded applicable government requirements. This case highlights the challenges that can arise when products are resold through multiple downstream parties beyond direct manufacturer control. It also underscores the importance of continued collaboration across industry and government to strengthen safeguards, enhance supply chain visibility, and facilitate the enforcement of export control laws. Supermicro will continue to cooperate with law enforcement and government officials in the United States, Taiwan and other jurisdictions to ensure our technology is distributed as lawfully intended."

The smuggling operation that led to arrests reportedly routed servers coming from Taiwan through Japan. Supermicro itself claims that the servers were sold to an authorized reseller, presumably in an attempt to wash its hands of any accusations that the company was directly involved.

Although three Supermicro employees have been charged by U.S. officials in connection to an earlier smuggling operation, neither Supermicro nor Nvidia have been charged. Despite this, Supermicro's stock plummeted nearly 30% after news of U.S. charges broke.

The U.S. government, and presumably Taiwanese authorities, have been cracking down on server smuggling to China over the past several months. In March, the Trump Administration proposed a strict licensing system for advanced AI hardware exports, and some U.S. senators have called on a wholesale ban on export licenses for Nvidia hardware to China, as well as several other south-east Asian countries including Malaysia, Thailand, and Vietnam.

Nvidia CEO Jensen Huang called on Super Micro Computer to strengthen its export compliance controls after arriving in Taipei on Saturday, months after U.S. federal prosecutors charged the server maker's co-founder and two others with conspiring to smuggle approximately $2.5 billion worth of Nvidia-equipped servers to China through shell companies in Southeast Asia.

Huang told reporters at Songshan Airport that Nvidia insists its partners follow U.S. trade rules. "We insist our partners are compliant. We hope that they will enhance and improve their regulation compliance and prevent that from happening in the future," Huang said in an address to the media.

Huang's comments came days after Taiwan launched its first formal crackdown on illicit AI hardware exports. The Keelung District Prosecutors' Office announced earlier this week that three suspects had submitted fraudulent shipping declarations to export Super Micro servers containing Nvidia AI chips to China, Hong Kong, and Macau.

The Taiwan case is separate from, but closely related to, the much larger U.S. federal prosecution unsealed in March. That indictment charged Supermicro co-founder Yih-Shyan "Wally" Liaw and two others with conspiring to smuggle approximately $2.5 billion worth of Nvidia-equipped servers to China through shell companies in Southeast Asia. Liaw has pleaded not guilty, and Supermicro has said it’s not named as a defendant and is cooperating with the investigation.

In the same press scrum at Songshan Airport, Huang confirmed that China is included in the $200 billion addressable market he projected for Nvidia's upcoming Vera CPU during the company's earnings call on May 20th. "H200 has been licensed to ship to China. It would be terrific to be able to serve that market. The Chinese market is very important. It's very large, of course," Huang told reporters, according to Reuters.

Despite the licensing approval, not a single H200 has been delivered to a Chinese customer. While roughly 10 Chinese firms have been cleared to purchase the chip, shipments haven’t started, and President Trump's talks with Chinese President Xi Jinping in Beijing earlier this month produced no breakthrough on Nvidia chip sales.

Huang is in Taipei ahead of Nvidia's GTC Taipei event and his Computex keynote on June 1st, where he’s expected to explore the Vera Rubin platform's software stack. He described the platform as "the largest product launch, probably in the history of Taiwan," noting that each Vera Rubin NVL72 system contains nearly 2 million parts and involves around 150 Taiwanese ecosystem partners.

Public documents reveal that four Chinese universities, two of which are known for conducting military research for the People’s Liberation Army, bought AI servers containing export-controlled Nvidia chips from Super Micro Computer. According to Reuters, the Harbin Institute of Technology (HIT), which focuses on missile, satellite, and robotics technologies, bought a system from Super Micro in July 2025 that contained eight Nvidia A100 AI GPUs.

Beihang University, which also conducts aerospace and defense research, is said to have acquired a machine-learning workstation from the company just this March, configured with four Nvidia A100 chips. These two universities are part of the “Seven Sons of National Defense,” a designation for Chinese academic institutions with deep ties to China’s Ministry of Industry and Information Technology and the People’s Republic of China's defense industry.

Nvidia’s A100 chips are less powerful than H200 AI GPUs, which U.S. President Donald Trump has finally allowed for export to China in late 2025. However, potential buyers still need to acquire export licenses from the federal government before they can get their hands on these chips, and it’s unlikely that PLA-linked institutions will ever get approval.

It’s unclear how the universities were able to purchase these Super Micro servers. However, three individuals, including Super Micro co-founder Yi-Shyan “Wally” Liaw, were recently arrested for smuggling $2.5 billion worth of advanced AI servers into China. The conspirators allegedly shipped orders from resellers and shell companies located in different countries in Southeast Asia. Once the servers containing the controlled AI GPUs arrived at the local warehouse, their serial numbers were transferred to dummy servers, and they were then issued new fake documentation. Once completed, they were forwarded to their final destination in China.

News like this has several American lawmakers on edge, with U.S. Senators Jim Banks (R-Ind.) and Elizabeth Warren (D-Mass.) writing a letter to Commerce Secretary Howard Lutnick urging him to suspend the issuance of export licenses to China and its intermediaries until the issue has been resolved. It would be a blow to Nvidia if the administration grants the request, especially as the company has finally received Chinese orders for its H200 chips after several months of uncertainty from Beijing.

Super Micro Computer, Inc. is facing a lawsuit from some shareholders who claim the company failed to disclose its dependence on illegal Chinese sales, resulting in an inflated stock price. According to Reuters, Super Micro is accused of committing securities fraud because its alleged illegal activities made it look like the firm had a stronger business outlook than actual. Furthermore, they said that the company failed to mention that it had issues with export controls compliance.

Three Super Micro employees, including its co-founder, Yih-Shyan “Wally” Liaw, were charged with conspiring to unlawfully divert cutting-edge U.S. artificial intelligence technology to China last week. It’s alleged that these illegal exports of high-end Nvidia chips delivered around $2.5 billion in sales for the company in 2024, which would account for more than 16% of the company’s total $14.94 billion in sales that year. This news led to a massive 33% drop in stock price for Super Micro, wiping out more than $6 billion in value for investors.

According to reports, real Super Micro servers were shipped to warehouses in Southeast Asia, where their serial numbers were removed and switched to dummy servers to fool inspectors. From there, the real AI servers were sent to China through a fake company that had fabricated paperwork. This shows that this was a deliberate, well-thought-out operation, with thousands of the fake servers still sitting across the region, supposedly awaiting delivery to local customers. While the federal government didn’t accuse Super Micro of any wrongdoing, the fact that a big chunk of its sales is tied to alleged illegal activity will hurt its stock price and shake investor confidence in the company.

This isn’t the only case that’s being prosecuted in the U.S. right now that relates to AI chip smuggling into China. Three individuals — one from China and two from the U.S. — have just been indicted over the same issue. The Department of Justice even released images of damning text messages between the conspirators, with one message composing a letter to encourage others to find clients who will “act as pass through (sic) partner for customers in China.” These cases, alongside the dozens of reports of AI chip smuggling from Singapore, Malaysia, and other countries near China, show that there is indeed a “healthy” black market for smuggled Nvidia chips.

President Donald Trump made a complete U-turn in late 2025 after he lifted the ban on H200 exports to China, meaning some of the chips that previously needed to be smuggled out of the U.S. could now be legally sold in the country. In fact, Nvidia CEO Jensen Huang said that the company has already received H200 orders from Chinese customers and that it has received licenses from the U.S. government to deliver them. Still, the uncertainty from Beijing has led some companies to consider purchasing these AI GPUs from the black market, especially as AI hyperscalers across the world race to build the most advanced model possible.

Three Super Micro employees have been charged with conspiring to unlawfully divert cutting-edge U.S. artificial intelligence technology to China. An indictment unsealed on Thursday says Yih-Shyan “Wally” Liaw, Ruei-Tsang “Steven” Chang, and Ting-Wei “Willy” Sun were involved in a smuggling plot that leveraged a middleman Southeast Asian company to fake paperwork and repackage servers that were powered by cutting-edge Nvidia chips. The underhanded operation is thought to have netted roughly $2.5b in sales since 2024.

Super Micro isn’t named in the indictment, but Liaw is a co-founder of the server firm, with nearly half a billion dollars worth of shares under his belt (the shares just dropped 12% in value on this news). Moreover, Chang is a sales manager for Super Micro in Taiwan. Sun is referred to as a third-party broker and fixer who has worked with the other two at Super Micro previously. Liaw (71, a U.S. citizen), and Sun (44, Taiwan) have been arrested, but Chang (53, Taiwan) is currently a fugitive, notes the Justice Department PR.

The transshipment scheme exposed

Basically, billions of dollars of Nvidia-powered servers that shouldn’t have been available to Chinese customers were funneled to the country using a fake front company, which fabricated paperwork and assembled thousands of dummy servers, to fool inspectors. It was far from a casual or opportunistic operation, with a high level of coordinated deception.

Despite the sophistication of the covert documentation, logistics, and supporting operations, some analysts have highlighted the brazen absurdity of the operation. For example, the indicted trio have been accused of maintaining a substantial inventory of thousands of dummy (empty shell) servers in Southeast Asia, supposedly awaiting deployment to local (not China-based) customers.

Analyst Max Weinbach highlights some CCTV image captures showing workers using hair dryers to transfer serial number stickers from genuine servers to the dummy server shells. They would subsequently ship the real GPU-packed servers to China.

Today, due to the ever-shifting sands of U.S. export controls policy, some of the smuggled Super Micro servers might be approved as exportable after following a framework to get a license.

Super Micro and Nvidia aren’t in the FBI’s crosshairs

Super Micro has acknowledged that the accused trio are “associated” with it, but highlights that the company isn’t named as a defendant in the indictment. The official statement says that it has “placed the two employees on administrative leave and terminated its relationship with the contractor, effective immediately.”

At the same time, the server maker distanced itself from the actions of the trio. The smuggling caper was described as “a contravention of the Company's policies,” and Super Micro “maintains a robust compliance program and is committed to full adherence to all applicable U.S. export and re-export control laws and regulations.”

Nvidia sent a similar themed statement to Tom’s Hardware. “Strict compliance is a top priority for Nvidia. We continue to work closely with our customers and the government on compliance programs as export regulations have expanded,” wrote the AI and graphics chip maker. “Unlawful diversion of controlled U.S. computers to China is a losing proposition across the board—Nvidia does not provide any service or support for such systems, and the enforcement mechanisms are rigorous and effective.”

Liaw, Chang, and Sun face some very serious federal felony charges. It is early days in the investigation, but it is possible that these people, in trying to enrich themselves, will end up serving decades (up to 30 years) behind bars, and/or fines, asset forfeiture, and also bans from employment in export-controlled industries.

Supermicro CEO Charles Liang announced on X (formerly Twitter) that he will set up shop near xAI’s Colossus AI supercomputer to help Musk achieve his dream of one million GPUs. Liang said that he plans to establish “local operations/production, validation, service and support.” Although it has been previously reported that Elon Musk shifted $6 billion worth of AI server orders from Supermicro to its competitors because of its financial problems, it seems xAI still plans to order from the beleaguered server supplier.

While Nvidia is likely the biggest winner behind Musk’s planned one million GPUs, which will most probably be powered by a mix of Hopper and Blackwell GPUs, it will also require all the accouterments needed to run a complete supercomputer. This includes servers and liquid-cooling solutions, which Supermicro is known for building. And with such a massive order, it does make sense for Liang and his company to put up local operations to support Musk’s ‘Gigafactory of Compute’.

This move will benefit xAI’s operations, as Supermicro would be able to deliver its products and services to such a massive AI server with much greater efficiency. But aside from that, it would also mean that the company would invest millions, if not billions, of dollars in Memphis and the surrounding area to get its operations up and running in support of the Colossus AI supercomputer.

An investment of this scale would greatly benefit the city and the state of Tennessee, bringing in a lot of money in investments and likely creating several thousand new jobs, both directly and indirectly. However, some of the surrounding communities are also concerned about the negative impact of having such a power-hungry industry setting up shop in their area.

Some experts estimate that the 100,000 H100 GPUs Musk currently has in his Memphis facility need 155MW to run. While it already has approval from the Tennessee Valley Authority (TVA) to receive 150MW, some community members are concerned that this huge demand will cause energy prices to spike for residential consumers and that the quick solution to this problem would be to build a new gas plant.

After all, if Musk brings an additional 900,000 GPUs online in the coming years, then his AI facility would likely require over 1.5GW of power. And if other corporations, like Supermicro, set up shop locally, they would need power, too, thus putting a lot of strain on the local electricity supply grid.

The massive power requirements of AI training have pushed several tech companies to invest in nuclear power. Amazon, Google, Microsoft, and Oracle have already signed contracts to develop small modular reactors or restart old large nuclear plants. Even Meta is getting in on the action, having recently released a request for proposals. However, it seems that Elon hasn’t made any plans to invest in nuclear power just yet.

This development is good news for the labor force in Memphis and the state of Tennessee, especially as it has the potential to bring in new jobs and investments in the state and city. However, it must also consider the impact of such a power-hungry industry entering the area.

xAI, Elon Musk’s AI startup, has shifted all AI server orders from troubled Supermicro to Dell, reports UDN.com. Dell, already among the largest makers of servers, reportedly benefits from this decision, just like its suppliers, Inventec and Wistron. In contrast, losing a multi-billion business to rivals amid a potential NASDAQ delisting could be another devastating blow for Supermicro.

Dell and Supermicro used to supply Musk’s companies, including xAI and Tesla. Musk even appeared publicly with Supermicro’s CEO, Charles Liang, who revealed that xAI had made substantial purchases of Supermicro’s liquid-cooled AI servers. However, after the U.S. Department of Justice began to probe Supermicro for accounting manipulations and alleged export violations to China and Russia and its stock plummeted 35% in one day, UDN says Musk’s companies decided to shift orders away from the troubled company.

Among the largest AI server suppliers, Dell is well-positioned to absorb orders. If true, Wistron, which produces motherboards for Dell’s AI servers and does some assembly tasks, would be one of the biggest beneficiaries of this shift. In fact, Wistron is already expanding its production capacity to meet surging AI demand, particularly in its three Hsinchu facilities in Taiwan and its Mexican operations. Wistron is optimistic about the growing demand for AI servers and aims for triple-digit annual growth.

Inventec, another major supplier for Dell, would also reap the rewards from an order realignment. Inventec has long been involved in AI server production and is one of Dell’s top three global server assembly partners. This year, the company primarily supplied machines based on Nvidia’s Hopper processors. UDN claims the company will be ready to mass-produce Nvidia Blackwell-based machines (powered by B200 and B200A GPUs) in the first quarter of 2025. The company reportedly has spare manufacturing capacity in Mexico, so it will likely be able to produce more AI servers for companies formerly served by Supermicro.

Supermicro’s issues stem from delayed financial filings, putting the company at risk of being delisted from NASDAQ. To avoid delisting, Supermicro needed to submit a plan by November 16 explaining the delay and specifying when the required 10-K annual report would be filed. Since the 16th fell on a Saturday, the company’s final deadline is November 18.

If Supermicro is delisted, it could face serious financial repercussions, including a sharp decline in stock value and the immediate repayment of $1.725 billion in convertible notes, which could severely harm the company.

As GPUs for AI and HPC applications get more power-hungry, they require better cooling. For example, a rack with 72 Nvidia B200 GPUs is projected to consume around 120 kW, which is 10 times higher than a typical rack power. As a result, Blackwell-based servers will mandate liquid cooling. On Monday, Supermicro introduced its datacenter-scale liquid cooling solution to prepare for these next-generation servers.

Supermicro's new complete system includes advanced cooling components, software, and management tools to optimize performance and sustainability. In particular, the solution incorporates Coolant Distribution Units (CDUs), cold plates, Coolant Distribution Manifolds (CDMs), cooling towers, and SuperCloud Composer software for end-to-end management. This setup is intended to lower initial investment costs and the total cost of ownership (TCO) by making datacenters more energy-efficient.

Supermicro's cooling technology is aimed at ultra-dense AI servers, each capable of housing dual top-tier CPUs and up to eight Nvidia HGX GPUs in a 4U configuration as well as up to 96 B200 GPUs per rack. This enables four times the computing density compared to traditional setups. 

One of the key benefits of Supermicro's liquid cooling solution is its rapid deployment capability. Supermicro's modular design reduces the time required to get datacenters up and running, cutting it down from months to weeks, which is crucial for companies looking to upgrade or build new facilities quickly. 

The solution’s liquid cooling capabilities are engineered to handle extreme heat levels, efficiently cooling servers that demand up to 12kW of power and AI racks generating over 100kW of heat, exactly what Blackwell-based NVL72 servers need. 

Supermicro's technology also supports warm water cooling, reaching temperatures up to 113°F (45°C). This feature not only boosts cooling efficiency but also makes it possible to repurpose the heat for applications like district heating or greenhouse warming. 

The SuperCloud Composer software included in the package offers real-time monitoring and control over all components. It helps data center operators manage costs and maintain system reliability, ensuring that operations run smoothly and efficiently. 

Supermicro said that it had recently implemented over 100,000 GPUs using its liquid cooling solution (DLC) in some of the largest AI facilities ever built, along with other cloud service providers (CSPs). 

"Supermicro continues to innovate, delivering full datacenter plug-and-play rack scale liquid cooling solutions," said Charles Liang, CEO and president of Supermicro. "Our complete liquid cooling solutions, including SuperCloud Composer for the entire life-cycle management of all components, are now cooling massive, state-of-the-art AI factories, reducing costs and improving performance. The combination of Supermicro deployment experience and delivering innovative technology is resulting in data center operators coming to Supermicro to meet their technical and financial goals for both the construction of greenfield sites and the modernization of existing data centers. Since Supermicro supplies all the components, the time to deployment and online are measured in weeks, not months."

Fujitsu is collaborating with Supermicro to build liquid-cooled servers by 2027, according to a report by The Register. These liquid cooler servers will be based on Fujitsu's upcoming ARM-based Monaka processor, which is slated to be released in the same timeframe.

The combination of liquid cooling and Fujitsu's energy-efficient Monaka chips is aimed at combating sky-high demand for data center capacity, which has become greater than what can be supplied thanks to various factors, including AI. One of the biggest obstacles preventing accelerated data center capacity development the ability to meet the growing power consumption of modern datacenter chips. By combining the efficiency of the ARM architecture with liquid cooling, Fujitsu and Supermicro hope to offer a market-leading server portfolio for their customers. 

Monaka is the name of Fujitsu's next generation ARM-based datacenter processor. The new chip is aimed at AI, HPC, and datacenter deployments featuring 150 Armv9-A cores with SVE2. Monaka is designed to take full advantage of the power efficiency of the ARM architecture, and Fujitsu has set an ambitious goal of having Monaka be twice as power-efficient as its competitors' chips — not its competitors' current chips, but those that will be made in 2026 and 2027. Monaka will be built on TSMC's 2nm fabrication process.

Fujitsu apparently originally designed these CPUs with air cooling in mind. However, the manufacturer is now shifting gears in this partnership with Supermicro. The main goal is to reduce the size of Monaka-based servers; liquid cooling paired with highly power-efficient processors allows designers to build highly compact cooling solutions. 

It's also likely that liquid cooling Monaka could result in greater power efficiency gains as compared to air cooling. Testing by SMC has revealed that Nvidia's GPU servers are 50% more power efficient when using submersion liquid cooling compared to air cooling. We don't know how exactly these servers will be setup, but Fujitsu and Supermicro have an opportunity to make some of the densest and most power-efficient servers in the world by 2027, if Fujitsu can deliver on its goals for Monaka. 

The U.S. Department of Justice (DoJ) is reportedly investigating Supermicro after Hindenburg Research accused the company of manipulating its financial reports and violations of U.S. export regulations to Russia and China, reports the Wall Street Journal. Supermicro has denied any wrongdoing, but the investigation is in its early stages, and stock prices have dropped in response. The DoJ and Supermicro have not formally confirmed the ongoing probe.

DoJ reportedly probes Supermicro

Hindenburg Research published a report titled Supermicro: Fresh Evidence Of Accounting Manipulation, Sibling Self-Dealing And Sanctions Evasion At This AI High Flyer back in August. The report accused Supermicro of improper accounting practices, related-party transactions, and failure to comply with U.S. export regulations. Supermicro is accused of selling high-tech products to China and Russia. This is after Russia invaded Ukraine in early 2022 and follows strict U.S.-imposed export controls on high-performance processors bound for China. The Hindenburg investigation appears comprehensive — the short seller investor says it lasted three months and included interviews with former Supermicro employees.  

Following Hindenburg's report, Supermicro postponed filing its annual report, stating that it needed to review its internal financial controls, which raised suspicions. Now, the Department of Justice has started its own inquiry into the company, focusing on allegations of financial misconduct. According to the report, a prosecutor contacted individuals with potential knowledge of these practices, including information about a former employee who accused Supermicro of violating accounting rules.

Dealing with Russia

Following the Russian invasion of Ukraine in 2022, the U.S. implemented strict export restrictions on high-performance computers and related technology to Russia. As a result, approximately 46 companies involved in handling these products are now under U.S. sanctions, with two-thirds of these exports consisting of components deemed critical by the U.S. government, potentially intended for military use. 

Despite Supermicro's claims of halting sales and recording no revenue from Russia since the war began, the company's exports to Russia have surged threefold, according to Hindenburg's findings. 

One significant recipient of Supermicro products is Niagara Computers, a supplier linked to a major Russian supercomputer used at a previously secret and now-sanctioned Kurchatov Institute nuclear technology research center. Niagara Computers received $46.3 million in products since the start of the war. Allegedly, these sales were initially facilitated through a California-based distributor but were later funneled through three new Turkish shell companies, one of which was sanctioned for smuggling.  

Additionally, around $30 million in components were allegedly shipped through a Hong Kong-based shell entity to VneshEcoStyle, one of Russia's largest importers of dual-use technology, which is also now under sanctions. This company makes no secret that its business is to 'find the equipment abroad and to deliver it in Russia,' which essentially means evading sanctions.

Trading with China

As the U.S. government has been cracking down on technology sales to China and Supermicro has come under scrutiny, the company's business practices to supply technology to Tianxia purportedly become more sophisticated.  

Hindenburg Research's report alleges that Supermicro exported AI, HPC, and surveillance servers equipped with Nvidia processors to China, even to entities with ties to the Chinese military through resellers. 

The report suggests that Supermicro employed intermediaries (for example, using Taiwan-based Leadtek, which derives about 70% - 80% of its revenue from China) and shell companies to obscure the final destinations of its products, thus evading U.S. export control regulations. It alleges that this practice enabled Supermicro to bypass restrictions and continue business in China despite tightening export laws. 

The report suggests that these actions demonstrate systemic governance failures within Supermicro and a willingness to circumvent trade restrictions to boost revenue and profits. This, of course, raises geopolitical concerns about the role of U.S. companies in advancing the technological capacity of strategic rivals. Hindenburg suggests that Supermicro potentially risked violating U.S. laws designed to prevent sensitive technologies from aiding foreign militaries. However, the report provides no direct evidence of Supermicro selling restricted components (Nvidia's A100 or H100 processors) to China. 

Stock drops 

Supermicro has benefited greatly from the rise of artificial intelligence, with its market value soaring from $4.4 billion in recent years to $67 billion by March 2024. The company is a major supplier of AI servers, which has driven much of its recent success.  

Despite Supermicro's growth, the report has cast doubt on its business practices. Apparently, just selling AI servers to interested parties was not enough. Based on Hindenburg's research, the company sold its servers to entities possibly linked to China and Russia's military amid the war against Ukraine, in which China aids Russia with technology. 

Supermicro's stock dropped by 12% after news of the DOJ investigation became public due to the WSJ report. The AI stock boom, which had significantly boosted the company's valuation, has cooled recently as investors adjust their expectations regarding returns on AI investments. However, when it comes to Supermicro, the company's financial results were allegedly boosted by illegal business practices and supplying U.S. adversaries. 

Sometimes, shortages of tiny components that can hardly be called 'high-tech' cost companies tens or hundreds of millions of dollars. This happened to Supermicro, which could not get enough quick couplings for liquid cooling systems, which made it postpone shipments of products and delayed $800 million of revenue from one quarter to another, reports Economic Daily News.

"Some key new components shortage delayed about $800 million of revenue shipments to July, which lowered our EPS for June and will be recognized in our September quarter," said Charles Liang, chief executive of Supermicro, at the company's earnings call last week (via SeekingAlpha).

Liquid cooling is vital for high-performance AI servers. A liquid-cooling system in AI servers typically comprises six components: cooling distribution units (CDUs), cold plates, cabinets, fan walls, coolant distribution manifolds (CDMs), and quick couplings. Quick couplings connect the coolant flow between the cold plate and the cooling distribution unit (CDU) and are essential for liquid-cooled AI machines.

Some quick couplings are prone to leakage, and their quality is crucial. Due to the high demand for AI servers, which require hundreds of these components per cabinet, the market has also seen a surge in demand for quick couplings. As a result, quick couplings have risen sharply, from $40 to $60 per unit, with buyers willing to pay even more to secure them. This price increase reflects increased demand and major challenges in meeting it due to a shortage of suppliers.

The market for quick couplings is controlled by seven companies, including two based in China. However, due to sanctions related to the U.S.-China tech conflict, Chinese companies face restrictions limiting their ability to supply these components, exacerbating the shortage.

Taiwanese firms such as Global Tek, Fositek, and Lotes are taking advantage of this situation by accelerating their production and certification processes. These companies see a significant opportunity to fill the gap left by the constrained Chinese suppliers and meet the growing demand.

Global Tek is actively working with partners in its Wuxi and Taoyuan plants, with samples currently undergoing testing. The company expects to see revenue from these efforts by the fourth quarter of 2024 or early next year.

Fositek, backed by its parent company Asia Vital Components, has also submitted samples for customer certification and focuses on developing quick couplings. Lotes, meanwhile, anticipates making progress by the end of the third quarter, aiming to capitalize on this high-demand market.

Elon Musk's Texas Tesla Gigafactory is expanding to contain an AI supercomputer cluster, and Supermicro's CEO is a big fan of the cooling solution. Charles Liang, founder and CEO of Supermicro, took to X (formerly Twitter) to celebrate Musk's use of Supermicro's liquid cooling technology for both Tesla's new cluster and xAI's similar supercomputer, which is also on the way.

Pictured together among server racks, Liang and Musk are looking to "lead the liquid cooling technology to large AI data centers." Liang estimates the impact of Musk leading the move to liquid cooling AI data centers "may lead to preserving 20 billion trees for our planet," obviously referring to the improvements that could be had if liquid cooling were adopted at all data centers worldwide.

AI data centers are well known for their massive power draws, and Supermicro hopes to reduce this strain by pushing liquid cooling. The company claims direct liquid cooling may offer up to an 89% reduction in electricity costs of cooling infrastructure compared to air cooling. 

In a previous Tweet, Liang clarified that Supermicro's goal is "to boost DLC [direct liquid cooling] adoption from <1% to 30%+ in a year." Musk is deploying Supermicro's cooling at a major scale for his Tesla Gigafactory supercomputer cluster. The new expansion to the existing Gigafactory will house 50,000 Nvidia GPUs and more Tesla AI hardware to train Tesla's Full Self Driving feature. 

The expansion is turning heads thanks to the supermassive fans under construction to chill the liquid cooling, which Musk also recently highlighted in an X post of his own (expand tweet below). Musk estimates the Gigafactory supercomputer will draw 130 megawatts on deployment, with growth up to 500MW expected after Tesla's proprietary AI hardware is also installed. Musk claims that the facility's construction is nearly complete, and it is planned to be ready for deployment in the next few months. 

Tesla's Gigafactory supercomputer cluster is not to be confused with Elon's other multi-billion dollar supercomputer cluster, the X/xAI supercomputer, which is also currently under construction. That's right: Elon Musk is building not one but two of the world's largest GPU-powered AI supercomputer clusters. The xAI supercomputer is a bit more well-known than Tesla's, with Musk already having ordered 100,000 of Nvidia's H100 GPUs. xAI will use its supercomputer to train GrokAI, X's quirky AI chatbot alternative that is available to X Premium subscribers. 

Also expected to be ready "within a few months," the xAI supercomputer will also be liquid-cooled by Supermicro and already has a planned upgrade path to 300,000 Nvidia B200 GPUs next summer. According to recent reports, getting the xAI cluster online is a slightly greater priority for Musk than Tesla, as Musk reportedly ordered Nvidia to ship thousands of GPUs originally ordered for Tesla to X instead in June. The move was reported to have delayed Tesla's supercomputer cluster's construction by months, but like so much Musk-centric news, exaggeration is highly likely.

Investigations have provided evidence that China-based organizations could still get their hands on sanctioned Nvidia GPUs as recently as Feb 28, 2024. According to Reuters, Chinese universities and research institutes may have sidestepped the sanctions on the most powerful GPUs by buying servers packing these powerful accelerators. However, an Nvidia spokesperson indicated that the products sold were likely to include stocks that had been previously exported to resellers in China.

Tender documents uncovered by Reuters reveal that ten Chinese entities acquired advanced Nvidia chips between Nov 20, 2023, and Feb 28, 2024, by simply ordering servers equipped with them. It is noted that Dell, Gigabyte, and Supermicro server products (including the sanctioned Nvidia GPUs) were sold in China after the Nov 17 expanded embargo, raising questions about sanctions-swerving. Remember, this sanctions policy adjustment even encompassed Nvidia GPUs like the GeForce RTX 4090 consumer graphics card.

Reuters found that the server resellers included 11 little-known Chinese retailers. The sale and purchase of the sanctioned powerful GPUs isn’t illegal in China. Thus, the big unanswered question here is whether the sanctioned GPUs were already in China before November 17, 2023.

In its statement to Reuters, Nvidia told the news organization that the tenders were for products already exported and widely available before Nov 17, 2023. Moreover, Nvidia supported its partners by claiming the documents “do not indicate that any of our partners violated the export control rules.” Nevertheless, Nvidia said it would investigate further.

Of course, it isn’t just Nvidia that will be policing potential sanctions breaches. The U.S. Bureau of Industry and Security monitors the movement of restricted chips and examines cases where sanctions may have been swerved.

Dell, Gigabyte, and Supermicro also responded to inquiries made by Reuters. Dell said it had seen no evidence of servers packing restricted chips being sent to China. It also stated that it would be willing to terminate relationships with resellers found to be breaking regulations and export controls. Gigabyte’s response wasn’t as detailed, with the firm simply stating it complies with Taiwanese laws and international regulations. Supermicro denied any knowledge of third-party exports or re-exports of systems without required licenses. Its U.S. legal firm also asserted that Supermicro goes above and beyond the letter of the law by being proactive about what customers do with their servers.

Earlier this week we saw indications that ample stocks of servers and sanctioned Nvidia H100 GPUs are available to Chinese customers. However, it is difficult to know whether some of these posted offers are clearing existing stocks, genuinely sanctions-busting, or just fraud attempts by scammers.

REGISTER NOW and watch on demand. 

The explosion of AI is further heightening demand for storage performance and capacity as organizations feed models and databases with unprecedented amounts of data, meaning the next generation of storage technologies will need to deliver even greater performance, density and capacity than ever before.

Supermicro's fourth annual Open Storage Summit brings together leading storage experts from across the industry including drive manufacturers, compute components manufacturers, software developers and of course Supermicro's industry leading system architects to discuss the latest in storage technologies and how they will solve tomorrow's data challenges from the data center right out to the intelligent edge.

This year’s Summit includes a roundtable keynote session followed by five focus sessions, with guests from the storage industry's leading players including Intel®, AMD, NVIDIA, Micron, Kioxia, Solidigm, and Samsung, as well as Supermicro’s storage software partners. 

New Innovations For Storage Performance

In a time in which pure processing power is game-changing, it’s important to continually reflect on current solutions and look for new ways to keep business players progressing through new levels.  Sometimes, that progress means stopping investment in an old version of that game and crafting a whole new open world instead.

In Session 2 of our 2023 Open Storage Summit, you will hear from NVIDIA on how they are helping organizations build whole new worlds in which to operate. Through the introduction of the third pillar of computing - the Data Processing Unit - DPUs join CPUs and GPUs to create a futuristic blue sky environment in which applications are accelerated well beyond the capabilities of CPUs alone.

This is particularly important in the frenetically growing AI market, in which lightning-fast storage processing time means that critical business initiatives make their way to the leaderboard instead of being relegated to game-over status. 

During this session, players in the audience will: 

• Discover the limitations inherent in traditional storage architectures
• Understand the advantages of GPUDirect storage and RDMA for AI
• Learn how the GPUDirect Storage and RDMA work at the rack-level to combine the resources of multiple systems into one massive compute cluster
• Uplevel their knowledge around how DPUs can effectively offload compute tasks to massively improve storage performance

Watch all of the webinars on demand

Register now for full access to the storage industry's leading online event to get the latest on key storage trends as well as exclusive look into the future of high performance storage from the most influential minds in the industry.

 Join the discussion here! 

Although we have been talking about Intel's Ponte Vecchio for over three years now, including pictures from the lab, few of us got to see Intel's Data Center GPU Max 1550 (a product based on the PVC design) in the flesh. That's because the chips were delayed repeatedly and not offered in server systems, but now that's changed, with Intel promising these 8-OAM systems will be available for its customers in July. At Computex, Supermicro decided to correct this wrong and demonstrated its X13 8U 8-way Ponte Vecchio GPU system for AI and HPC as well as the compute GPU itself.

Intel's Ponte Vecchio represents the pinnacle of processor complexity, boasting a transistor count of over 100 billion (excluding memory) and a combined die size of 2,330 mm². The PVC design is a complex arrangement of 47 tiles that includes compute tiles, Rambo cache tiles, Xe Link tiles, and HBM2E memory stacks interconnected using Intel's advanced packaging technologies like Embedded Multi-Die Interconnect Bridge (EMIB) and Foveros. Tiles for PVC are made using a variety of process technologies and then assembled together.  

Supermicro's X13 8U 8-way Ponte Vecchio GPU server itself, it is based on two 4th Generation Xeon Scalable 'Sapphire Rapids' processors rated for up to 350W that are mated with 32 DIMM slots as well as up to 20 hot-swap 2.5-inch SSDs (12 NVMe, 8 SATA).

But while Intel's Ponte Vecchio is extremely complex inside, it looks pretty humble in its OAM form-factor with a heatspreader on. Given the fact that the 100-billion-transistors beast is rated for a 600W TDP, it has an intricated voltage regulating module that includes two high-performance converters each of which is serving its own 'half' of MOSFETs for added granularity of power delivery. By contrast, AMD's Instinct MI250X uses only one high-performance converter onboard, though it is 'only' rated for a 550W TDP. Meanwhile, an OAM slot can supply up to 700W of power, so Intel's module was most likely architected with headroom in mind. 

600W is an exceptional amount of power, so Supermicro's SYS-821PV-TNR uses huge radiators and high-pressure fans to remove the heat from eight of Intel's Data Center GPU Max 1550 processors. These radiators have seven copper heat pipes and dozens of aluminum fins, so even a PVC radiator is a quite an intricate arrangement. 

When we first saw mention of the Supermicro X13SWA-TF motherboard based on Intel’s W790 chipset set to power upcoming workstations as well as high-end desktops based on Sapphire Rapids CPUs, we barely had any details about it. Now, thanks to our colleagues from Chiphell who revealed its picture, we can tell more.

Aas mentioned, the Supermicro X13SWA-TF motherboard is based on W790, a workstation-grade chipset. So it comes in an E-ATX form factor, which is used for workstations, desktops, and tower servers. Speaking of workstations and tower servers, the platform indeed has an Aspeed remote management processor.

One of the first things that strikes the eye is an eight-phase solid-state voltage regulating module for Intel’s LGA4677 processor. To feed the beast, there are two EPS connectors. While the number of phases may not exactly impress overclockers, we are talking about an ultra-high-end product aimed mostly at workstation users who utmost value stability and longevity. For the same reason, the board features a four-phase VRM for its eight-channel memory subsystem.

Being a motherboard aimed at demanding users, the Supermicro X13SWA-TF also features four M.2-2280 slots for SSDs fed directly from the CPU. The board also brings six PCIe x16 slots, five of which are connected to the CPU and therefore support the PCIe 5.0 protocol. Meanwhile, there is no word about exact bifurcation or CXL support, though. Also, there are two U.2 connectors and a multitude of SATA connectors for bulk storage.

Since the Supermicro X13SWA-TF is aimed at a variety of applications, including desktop workstations, it also features an audio codec and appropriate 3.5-mm connectors. 

We know from an early leak that the X13SWA-TF is going to be priced in the ballpark of ultra-high-end enthusiast-grade motherboards, yet the mystery remains when exactly this one will hit the market. For those who have been waiting — considering that now both AMD and Intel have given up on so-called extreme workstations — availability might be the most important question.

A Canadian retailer has listed Supermicro's yet-to-be-announced X13SWA-TF motherboard based on the Intel W790 chipset. It is designed for workstations featuring Intel's Xeon processors in LGA4677 packaging, which points to Sapphire Rapids CPUs.

The Supermicro X13SWA-TF motherboard (listed at Atic.ca and spotted by @momomo_us) comes in an E-ATX form factor, which is used for workstations, desktops, and tower servers. The motherboard is priced at CAD$1290  ($965) with a discount if you pay cash, but there is no mention of its availability timeframe.

Earlier this year, Intel confirmed that its W790 is a workstation-grade chipset (not an entry-level workstation-grade chipset) but never disclosed which processors it supports. But since the motherboard has an LGA4677 socket, it should support Intel's Xeon processors codenamed Sapphire Rapids. 

We can speculate that we might be talking about a special version of Sapphire Rapids CPUs tailored for workstations (i.e., featuring fewer than 60 cores but with higher clocks) or Intel's 34-core Raptor Lake-S processors spotted by our own Paul Alcorn earlier this year. Meanwhile, both Sapphire Rapids and Raptor Lake-S feature high-performance cores based on the Golden Cove microarchitecture and have several things in common. Therefore, it is plausible to assume that Intel's 34-core Raptor Lake processors (also known as RPLS-34C) will be marketed under the Xeon brand, though Intel has never discussed it publicly. 

The Supermicro X13SWA-TF platform has six PCIe 4.0 x16 slots, two 10GbE ports controlled by Intel's X550 chip, and Intelligent Platform Management Interface (IPMI) for remote management, which is handy both for workstations and servers. The basic description does not mention PCIe 5.0 support or M.2 slots (indispensable for desktop PCs), but since this is an early representation of the X13SWA-TF's capabilities, this is not surprising. 

Rumors about a Sapphire Rapids-based platform for high-end desktops and workstations have been floating around for years. However, Intel has never publicly confirmed plans to address the client PC market with its SPR processors as it is focused on launching Sapphire Rapids for servers first. The listing is essentially the first confirmation that Intel's W790 platform supports the company's processors in LGA4677 packaging, which points to Intel Xeon 'Sapphire Rapids' CPUs. Meanwhile, Intel's 34-core Raptor Lake-S processors are perhaps better candidates for mainstream workstations and high-end desktops.

The push for enhanced compute density continues, and servers like Supermicro's 1024US-TRT, which hails from the company's 'A+ Ultra' family, are designed to answer that call with generous compute capabilities paired with copious connectivity options. Supermicro designed this slim 1U dual-socket server for high-density environments in enterprise applications, high-end cloud computing, virtualization, and technical computing workloads. The platform supports up to 8TB of DDR4 memory spread across 32 DIMMs along with plenty of PCIe 4.0 connectivity, dual 10 GbE LAN ports, and up to four NVMe devices in the front bays.

The platform supports dual AMD EPYC 7003 and 7002 processors, meaning it supports up to 64 cores and 128 threads with the EPYC Milan, Milan-X, or EPYC Rome processors. Supermicro's Intel Ice Lake X12 servers can't match that number of cores and threads in a single platform, signifying that the 1024US-TRT offers the utmost density in its portfolio. Naturally, Supermicro competes with other OEM server vendors, like Lenovo, Dell/EMC, and HPE, in the high-volume general-purpose 1U realm with the 1024US-TRT. 

AMD's EPYC Genoa will launch later this year to compete with Intel's incessantly-delayed Sapphire Rapids, setting the stage for either AMD's continued dominance or an Intel resurgence. As we wait for those launches, here's a look at some of our benchmarks and the current state of the data center CPU performance hierarchy in several hotly-contested price ranges. 

Supermicro 1024US-TRT Server

The Supermicro 1024US-TRT server comes in the 1U form factor, enabling incredible density. The server supports AMD's EPYC 7002 and 7003 processors that top out at 64 cores apiece, translating to 128 cores and 256 threads spread across the dual sockets. In addition, the platform also supports AMD's Milan-X chips (BIOS version 2.3 or newer) that come with up to 64 cores and 128 threads paired with a once-unthinkable 768MB of L3 cache. These chips help gear the 1024US-TRT for more diverse workloads beyond its traditional target markets, expanding to technical computing workloads, too. This includes workloads like Electronic Design Automation (EDA), Computational Fluid Dynamics (CFD), Finite Element Analysis (FEM), and structural analysis.

The 1024US-TRT has a tool-less rail mounting system with square pegs that eases installation into server racks, and the CSE-819UTS-R1K02P-A chassis measures 1.7 x 17.2 x 29 inches and slides into 19" racks.

The server accommodates CPU TDPs that stretch up to 280W, but using chips beyond 225W requires special accommodations. As such, the server can technically support the most powerful EPYC processors, like the 7Hxx-series models, but you'll need to verify those configurations with Supermicro. 

The front panel comes with standard indicator lights, like a color-coded information light that indicates various types of failures and overheating while also serving as a unit identification LED. It also includes hard drive activity, system power, and two LAN activity LEDs. Power, reset, and unit identification (UID) buttons are also present at the upper right of the front panel, with the latter illuminating a light on the rear of the server for easy location of the unit in a packed rack.

By default, the system has four tool-less 3.5-inch hot-swap SATA 3 drive bays, but you can configure the server to accept four NVMe drives on the front panel (you use 2.5" adaptors for the SSDs). We tested with a PCIe 4.0 Kioxia 1.92TB KCD6XLUL1T92 SSD. You can also add an optional SAS card to enable support for SAS storage devices and an optical drive. The front of the system also houses a slide-out service/asset tag identifier card to the upper left. This is important as it holds the default BMC user ID and password, enabling access to the remote management features. 

Popping the top off the chassis reveals two housings that hold fans. A total of eight fans feed air to the system, and each housing includes four Sunon 23,300 RPM counter-rotating 40 x 40 x 56 mm fans for maximum static pressure and reduced vibration. As expected with servers intended for 24/7 operation, the system can continue to function in the event of a fan failure. However, the remainder of the fans will automatically run at full speed if the system detects a failure. Naturally, these fans are loud, but that's not a concern. You manage the fan speed and profiles via the BMC (not the BIOS).

Four fans cool each CPU, and a simple black plastic shroud directs air to the heatsinks underneath. Dual SP3 sockets house the two processors, which are covered by standard CPU heatsinks optimized for linear airflow. 

A total of 16 memory slots flank each processor, for a total of 32 slots that support up to an incredible 8TB of ECC DDR4-3200 memory (via 256GB DIMMs), easily outstripping the memory capacity available with competing Intel platforms. We tested the EPYC Milan processor with 16x 16GB DDR4-3200 SK hynix modules for a total memory capacity of 256GB. In contrast, the Icel Lake Xeon comparison platform came with 16x 32GB SK hynix ECC DDR4-3200 for a total capacity of 512GB of memory. 

The H12DSU-iN motherboard's expansion slots consist of two full-height 9.5-inch PCIe 4.0 x16 slots and one low-profile PCIe 4.0 x16 slot, all mounted on riser cards. An additional internal PCIe 4.0 x16 slot is also available, but this slot only accepts proprietary Supermicro cards.

The rear I/O panel includes two 10 gigabit RJ45 LAN ports powered by an Intel X710-AT2 NIC, along with a dedicated RJ45 IPMI LAN port for management. Here we find the only USB ports on the machine, which come in the form of two USB 3.0 headers (it's a pity there isn't a USB port on the front), along with a COM and VGA port. 

Two 1000W 80% Plus Titanium-Level redundant power supplies with PMBus provide power to the server, with automatic failover in the event of a failure and hot-swapability for easy servicing. 

The BIOS is easy to access and use and offers plenty of tunable parameters, including CPU power threshold adjustments, while the IPMI web interface provides a wealth of monitoring capabilities and easy remote management that matches the type of functionality available with other types of platforms. Among many options, you can update the BIOS, use the KVM-over-LAN remote console, monitor power consumption, access health event logs, monitor and adjust fan speeds, and monitor the CPU, DIMM, and chipset temperatures and voltages. Supermicro's remote management suite is polished and easy to use, and we can say the same about the BIOS.

Test Setup

Bear in mind that the 240W 64-core EPYC 7713s we're testing aren't the highest-frequency 64-core in AMD's EPYC Milan arsenal. That distinction falls to the 64-core EPYC 7763 with a 2.45 base and 3.5 GHz boost clock rate, along with a much heftier 280W TDP. Keep that in mind when you compare this EPYC chip to the Ice Lake flagship, the Xeon Platinum 8380, which also comes with a much higher 270W TDP. 

We used the Supermicro 1024US-TRT server to test the EPYC 7713 Milan processors, while we used the Supermicro 1024US-TR4 server to test four different EPYC Rome configurations.

We used the Intel 2U Server System S2W3SIL4Q Software Development Platform with the Coyote Pass server board to test the Ice Lake Xeon Platinum 8380 processors. This system is designed primarily for validation purposes, so it doesn't have too many noteworthy features. However, the system is heavily optimized for airflow, with the eight 2.5" storage bays flanked by large empty bays that allow for plenty of air intake. You can read more about this configuration in our Intel Ice Lake Xeon Platinum 8380 Review. We used a Dell/EMC PowerEdge R460 server to test the other Xeon processors in our test group.

Our test configurations don't have balanced memory capacities, but that comes as an unavoidable side effect of the capabilities of each platform and the systems we've been sampled. As such, bear in mind that memory capacity disparities may impact the results below.  

We used the Phoronix Test Suite for testing. This automated test suite simplifies running complex benchmarks in the Linux environment. The test suite is maintained by Phoronix, and it installs all needed dependencies, and the test library includes 450 benchmarks and 100 test suites (and counting). Phoronix also maintains openbenchmarking.org, which is an online repository for uploading test results into a centralized database. We used Ubuntu 20.04 LTS and the default Phoronix test configurations with the GCC compiler for all tests below. We also tested both platforms with all available security mitigations. 

Linux Kernel and LLVM Compilation Benchmarks

The dual EPYC 7713 processors in the 1024US-TRT complete the Timed Linux Kernel Compilation benchmark, which builds the Linux kernel at default settings, in a mere 19 seconds, edging out the Xeon Platinum 8380s. That's incredibly impressive — remember, the 7713s operate at a 225W TDP, meaning it is the lower-frequency 64-core EPYC variant. In contrast, the Ice Lake chips have a 270W TDP rating. AMD's Milan flagship, the 7763, would take the lead in this benchmark. 

AMD's EPYC 7713 also carves out a win against the 8380s in the Timed LLVM Compilation workload, completing the test in 178 seconds. Additionally, performance scaling is more pronounced than we expected moving from the EPYC 7742 to the 7713 given that the latter is the low-frequency variant, with the 7713s completing the benchmark 10% faster than the previous-gen flagship. 

Molecular Dynamics and Parallel Compute Benchmarks

NAMD is a parallel molecular dynamics code designed to scale well with additional compute resources; it scales up to 500,000 cores and is one of the premier benchmarks used to quantify performance with simulation code. The EPYC processors are well-suited for these types of highly-parallelized workloads due to their prodigious core counts, but Intel's Ice Lake is also a force to be reckoned with. This workload fully saturates the cores with an extended workload, thus generating a heavy thermal load, but the Supermicro 1024US-TRT didn't show any signs of thermals impacting performance. Once again, the EPYC 7713 impresses against Intel's much higher-TDP flagship.

Stockfish is a chess engine designed for the utmost in scalability across increased core counts — it can scale up to 512 threads. Here we can see that this massively parallel code scales incredibly well, with the EPYC 7713s taking the top of the benchmark charts.   

Rendering Benchmarks

Provided you can keep the cores fed with data, most modern rendering applications also fully utilize the compute resources. The 64-core EPYC Milan and EPYC Rome processors lead this series of benchmarks convincingly, though Intel's Ice Lake does narrow the competition in a few of the Blender renders and leads in others.

Encoders tend to present a different challenge: As we can see with the VP9 libvpx benchmark, they often don't scale well with increased core counts. Instead, they often benefit from per-core performance and other factors, like cache capacity.  Here we can see that Intel's 8380 takes the lead by slim margins, but AMD's EPYC Milan 7713 is incredibly impressive given its much lower power and thermal budget. 

Compression and Security

Compression workloads also come in many flavors. For example, the 7-Zip (p7zip) benchmark exposes the heights of theoretical compression performance because it runs directly from main memory, allowing both memory throughput and core counts to impact performance heavily. Here we can see that benefit the EPYC 7713s tremendously as they take the lead over the 8380s. In contrast, the gzip benchmark, which compresses two copies of the Linux 4.13 kernel source tree, tends to respond well to speedy clock rates, giving the 16-core EPYC 7F52s the lead. The 7713s are still impressive, though, as they take second place. 

The open-source OpenSSL toolkit uses SSL and TLS protocols to measure RSA 4096-bit performance. As we can see, this test favors the EPYC processors due to its parallelized nature, but offloading this type of workload to dedicated accelerators is becoming more common for environments with heavy requirements.  

Conclusion

AMD's forward-thinking SP3 socket design has given server builders plenty of flexibility as the EPYC lineup matures, and that capability is on full display with the Supermicro 1024US-TRT server. With support for EPYC 7002 and 7003 processors, high-frequency H- and F-Series, and even the Milan-X CPUs, the platform has continued to evolve to support the growing EPYC roster. 

Supermicro's fully-validated systems come with the hardware fully configured, and rack installation with the tool-less rail kit is simple and quick. We found the system to be robust during our tests, with plenty of power for the processors fed by the high-quality Titanium power supplies and ample cooling provided by eight fans. We didn't notice any adverse impact from thermal generation during a bevy of extended-duration all-core workloads that generate plenty of thermal load, signifying that the cooling system is well designed. 

We'd like to see a few USB ports added to the front panel, as this eases setup and maintenance. That said, the easily-accessible IPMI interface provides comprehensive monitoring and maintenance options, not to mention a polished remote management interface. On the hardware side, the dual 10GbE LAN and PCIe 4.0 interfaces provide plenty of connectivity options, while the support for up to 8TB of ECC memory allows the server to exploit EPYC Milan's tremendous throughput while providing enough capacity to address even the most memory-hungry applications. The server's slim 1U chassis does limit you to four drive bays, but Supermicro supports plenty of high-capacity storage options to maximize the available bays.

The Supermicro 1024US-TRT server performed well in our tests, packing quite the threaded heft into a small form factor that can address general-purpose, enterprise, cloud, and virtualization roles well. The addition of Milan-X support also positions the platform for possible use in more diverse technical computing workloads, positioning the 1024US-TRT as a robust server fit for a wide variety of roles. 

• MORE: CPU Benchmark Hierarchy
• MORE: AMD vs Intel
• MORE: How to Overclock a CPU

Nvidia's decision to build its own lineup of Arm-based server CPUs, dubbed the Grace CPU Superchip and the Grace Hopper Superchip, charted the company's course to building out full systems with its CPUs and GPUs in the same box. That initiative moved closer to reality today as Nvidia announced at Computex 2022 that several major server OEMs will offer dozens of reference systems based on its new Arm CPUs and Hopper GPUs in the first half of 2023.

Nvidia tells us that these new systems, which we break down below, will co-exist with its existing lineup of reference servers, so the company will continue to support x86 processors from AMD and Intel for the foreseeable future. 

As a quick reminder, Nvidia's Grace CPU Superchip is the company's first CPU-only Arm chip designed for the data center and comes as two chips on one motherboard, while the Grace Hopper Superchip combines a Hopper GPU and the Grace CPU on the same board. The Neoverse-based CPUs support the Arm v9 instruction set and systems come with two chips fused together with Nvidia's newly branded NVLink-C2C interconnect tech.

Overall, Nvidia claims the Grace CPU Superchip will be the fastest processor on the market when it ships in early 2023 for a wide range of applications, like hyperscale computing, data analytics, and scientific computing. (You can read the deep-dive details about the silicon here.)

A slew of blue-chip OEM/ODMs, like Asus, Gigabyte, Supermicro, QCT, Wiwynn, and Foxconn, have dozens of new reference server designs planned for launch in the first half of 2023, indicating that Nvidia remains on track with its Grace CPU silicon. The OEMs will craft each of the server designs from one of Nvidia's four reference designs that include server and baseboard blueprints. These servers will be available in 1U and 2U form factors, with the former requiring liquid cooling.  

The Nvidia CGX system for cloud graphics and gaming applications comes with the dual-CPU Grace Superchip paired with Nvidia's A16 GPUs. The Nvidia OVX servers are designed for digital twin and omniverse applications and also come with the dual-CPU Grace, but they allow for more flexible pairings with many different Nvidia GPU models. 

The Nvidia HGX platform comes in two flavors. The first is designed for HPC workloads and only comes with the dual-CPU Grace, no GPUs, and OEM-defined I/O options. Meanwhile, on the far right, we see the more full-featured HGX system for AI training, inference, and HPC workloads with the Grace CPU + Hopper GPU Superchip, OEM-defined I/O, and support for the option of fourth-gen NVLink for connections outside of the server via NVLink switches.   

Notably, Nvidia will offer the NVLink option with its CPU+GPU Grace Hopper Superchip models, but not for systems powered by the dual-CPU Grace Superchip. 

Here we can see the two different 2U blades that can power the HGX systems. The dual-CPU 'HGX Grace' CPU Superchip blade has up to 1TB of LPDDR5x memory, provides up to 1TB/s of memory bandwidth, adheres to a 500W TDP envelope, can be cooled with either liquid or air, and supports two blades per node for up to 84 nodes per rack. 

The HGX 'Grace Hopper' Superchip blade comes with a single Grace CPU paired with the Hopper GPU, providing 512 GB of LPDDR5x memory, 80GB of HBM3, and up to a combined 3.5 TB/s of memory throughput. As you'd expect, given the addition of the GPU, this blade comes with a higher 1000W TDP envelope and comes with either air or liquid cooling. This larger blade limits the HGX Grace Hopper systems to 42 nodes per rack. 

Unsurprisingly, Nvidia offers all of these systems with its other important additive that is helping the company become a solutions provider: The 400 Gbps Bluefield-3 Data Processing Units (DPUs) that come as the fruits of its Mellanox acquisition. These chips offload critical work from the CPUs, allowing streamlined networking, security, storage, and virtualization/orchestration features.

Nvidia already has CGX, OVX, and HGX systems available with x86 CPUs from both Intel and AMD, and the company tells us that it plans to continue to provide those servers and develop newer revisions with Intel and AMD silicon.

"x86 is a very important CPU that is pretty much all of the market of Nvidia GPUs today. We'll continue to support x86, and we'll continue to support Arm-based CPUs, offering our customers in the market the choice of wherever they want to deploy accelerated computing," Paresh Kharya, Nvidia's Senior Director of Product Management and Marketing, told us. 

That doesn't mean that Nvidia will pull performance punches, though. The company recently demoed its Grace CPU in a weather forecasting head-to-head with Intel's Ice Lake, claiming that its Arm chip is 2X faster and 2.3X more efficient. Those same Intel chips currently power Nvidia's OVX servers. 

The company hasn't spared AMD, either. Nvidia also claims that its Grace CPU Superchip is 1.5X faster in the SPECrate_2017_int_base benchmark than the two previous-gen 64-core EPYC Rome 7742 processors it uses in its current DGX A100 systems.

We'll soon see if Nvidia's first foray into CPUs lives up to its claims. The reference CGX, OVX, and HGX systems will ship in the first half of 2023. 

Multiple European retailers have begun listing Supermicro's MBD-X13SAE-series motherboards based on Intel's yet-to-be-announced W680 chipset. That chipset is designed for upcoming Intel's Xeon E-3300-series 'Alder Lake' CPUs in LGA1700 packaging. The listings imply that Intel's next-generation entry-level Xeon E platform for entry-level servers and workstations is nearing its launch. But some retailers will sell you the motherboards today.

Supermicro's MBD-X13SAE-series family will include at least two Intel W680-based models: the MBD-X13SAE and the MBD-X13SAE-F, according to a listing at Sona.de (spotted by Twitter leaker @momomo_us). Both motherboards support all of the processors in Intel's LGA1700 form-factor, including the unannounced Xeon E-3300-series CPUs featuring the hybrid Alder Lake architecture, and are equipped with four memory slots supporting DDR5 memory modules with or without ECC (according to Ahead-IT.eu). Also, the new platforms naturally support PCIe 5.0 as well as other advantages that Alder Lake has to offer. 

Traditionally, memory link ECC (do not confuse with DDR5's on-die ECC technology that is implemented to improve DRAM IC yields) is one of the key features that differentiates Intel's Xeon E-series CPUs that support memory modules with from similarly configured Core processors that do not support ECC capability. That said, Intel's E-3300-series CPUs and Intel's W680 platform will require DDR5 memory modules with ECC to show one of its key advantages over desktop-oriented platforms. 

But while the Intel's Xeon E-3300-series processors and Intel's W680 platform are getting closer, we still have little idea about what to expect from new CPUs and chipset. Early leaks indicated that the new entry-level server/workstation platform will offer up to 16 CPU cores with TDPs of 35 W, 65 W and 125 W. Meanwhile, we have no idea whether Intel intends to enable AVX-512 support on its upcoming Xeon E-3300-series processors, or will keep it disabled, like it does with 12th Generation Core CPUs. 

As far as availability is concerned, Supermicro's MBD-X13SAE-series motherboards based on Intel's W680 chipset can be ordered today from several European retailers. Two Intel W680-based platforms — the MBD-X13SAE (priced at €442 with VAT or $425 without tax) and the MBD-X13SAE-F (priced at €492 with VAT or $473 without tax) — can be ordered from Sona.de or CTT.de and delivered in 3 – 5 business days. The Netherlands-based Ahead-IT.EU promises to ship these motherboards in 5 – 7 days, whereas Italy-based IsyCorp says that these products are available from the manufacturer. 

Given that so many retailers are about to ship their Intel W680 motherboards, it looks like the launch of Intel's next-generation Xeon E-3300-series platform is imminent.

Thanks to Anandtech, we have information on two new Supermicro motherboards designed for Intel's new Ice Lake Xeon Scalable CPUs: the 12DPL-NT6 and X12DPL-i6. The boards feature dual LGA4189 sockets -- for a max potential configuration of 80 CPU cores all squished into a standard ATX form factor.

The stand-out feature of the X12DPL-NT6 and X12DPL-i6 is the ATX form factor; Supermicro has demonstrated it can put two gigantic LGA4189 sockets onto an ATX board without sacrificing many features; the two sockets alone take up nearly half of the entire board's size.

The only feature Supermicro had cut out was the platform's maximum support of twelve DIMM slots. Due to the size constraints, each CPU can only access up to four DIMM slots (eight total on the board), meaning each chip will be limited to quad-channel memory configurations. This will only be a problem if you're workloads benefit from significantly high memory bandwidth/capacity.

For connectivity, you get four PCIe 4.0 slots on the bottom, with each slot having the full 16 lanes available. For storage, the boards support twelve SATA slots with RAID 0, 1, 5, and 10 array support, and dual M.2 PCIe Gen 4.0 slots.

The only difference between the X12DPL-NT6 and X12DPL-i6 is the ethernet and M.2 configuration. With the X12DPL-NT6 you get dual Intel X550 10Gb Ethernet ports, and one more M.2 slot capable of PCIe Gen 4.0 x8 support.

The lower end X12DPL-i6 does not include 10Gb Ethernet and instead relies on two Intel i210 Gigabit LAN controllers for network connectivity. It also loses the x8 M.2 slot.

We have no idea of when these boards will be available to purchase, but if you're in the market for something like this, Supermicro will probably happily return your emails or phone calls.

Supermicro's 1023US-TR4 is a slim 1U dual-socket server designed for high-density compute environments in high-end cloud computing, virtualization, and enterprise applications. With support for AMD's EPYC 7001 and 7002 processors, this high-end server packs up to two 64-core Eypc Rome processors, allowing it to cram 128 cores and 256 threads into one slim chassis. 

We're on the cusp of Intel's Ice Lake and AMD's EPYC Milan launches, which promise to reignite the fierce competition between the long-time x86 rivals. In preparation for the new launches, we've been working on a new set of benchmarks for our server testing, and that's given us a pretty good look at the state of the server market as it stands today. 

We used the Supermicro 1023US-TR4 server for EPYC Rome testing, and we'll focus on examining the platform in this article. Naturally, we'll add in Ice Lake and EPYC Milan testing as soon as those chips are available. In the meantime, here's a look at some of our new benchmarks and the current state of the data center CPU performance hierarchy in several hotly-contested price ranges. 

Inside the Supermicro 1023US-TR4 Server

The Supermicro 1023US-TR4 server comes in the slim 1U form factor. And despite its slim stature, it can host an incredible amount of compute horsepower under the hood. The server supports AMD's EPYC 7001 and 7002 series chips, with the latter series topping out at 64 cores apiece, which translates to 128 cores and 256 threads spread across the dual sockets.

Support for the 7002 series chips requires a 2.x board revision, and the server can accommodate CPU cTDP's up to 280W. That means it can accommodate the beefiest of EPYC chips, which currently comes in the form of the 280W 64-core EPYC 7H12 with a 280W TDP. 

The server has a tool-less rail mounting system that eases installation into server racks and the CSE-819UTS-R1K02P-T chassis measures 1.7 x 17.2 x 29 inches, ensuring broad compatibility with standard 19-inch server racks. 

The front panel comes with standard indicator lights, like a unit identification (UID) light that helps with locating the server in a rack, along with drive activity, power, status light (to indicate fan failures or system overheating), and two LAN activity LEDs. Power and reset buttons are also present at the upper right of the front panel.

By default, the system comes with four tool-less 3.5-inch hot-swap SATA 3 drive bays, but you can configure the server to accept four NVMe drives on the front panel, and an additional two M.2 drives internally. You can also add an optional SAS card to enable support for SAS storage devices. The front of the system also houses a slide-out service/asset tag identifier card to the upper left. 

Popping the top off the chassis reveals two shrouds that direct air from the two rows of hot-swappable fans. A total of eight fan housings feed air to the system, and each housing includes two counter-rotating 4cm fans for maximum static pressure and reduced vibration. As expected with servers intended for 24/7 operation, the system can continue to function in the event of a fan failure. However, the remainder of the fans will automatically run at full speed if the system detects a failure. Naturally, these fans are loud, but that's not a concern for a server environment.  

Two fan housings are assigned to cool each CPU, and a simple black plastic shroud directs air to the heatsinks underneath. Dual SP3 sockets house both processors, and they're covered by standard heatsinks that are optimized for linear airflow. 

A total of 16 memory slots flank each processor, for a total of 32 memory slots that support up to 4TB of registered ECC DDR4-2666 with EPYC 7001 processors, or an incredible 8TB of ECC DDR4-3200 memory (via 256GB DIMMs) with the 7002 models, easily outstripping the memory capacity available with competing Intel platforms.

We tested the EPYC processors with 16x 32GB DDR4-3200 Samsung modules for a total memory capacity of 512GB. In contrast, we loaded down the Xeon comparison platform with 12x 32GB Sk hynix DDR4-2933 modules, for a total capacity of 384GB of memory. 

The H11DSU-iN motherboard's expansion slots consist of two full-height 9.5-inch PCIe 3.0 slots and one low-profile PCIe 3.0 x8 slot, all mounted on riser cards. An additional internal PCIe 3.0 x8 slot is also available, but this slot only accepts proprietary Supermicro RAID cards. All told, the system exposes a total of 64 lanes (16 via NVMe storage devices) to the user. 

As one would imagine, Supermicro has other server offerings that expose more of EPYCs available 128 lanes to the user and also come with the faster PCIe 4.0 interface. 

The rear I/O panel includes four gigabit RJ45 LAN ports powered by an Intel i350-AM4 controller, along with a dedicated IPMI port for management. Here we find the only USB ports on the machine, which come in the form of two USB 3.0 headers, along with a COM and VGA port. 

Two 1000W Titanium-Level (96%+) redundant power supplies provide power to the server, with automatic failover in the event of a failure, as well as hot-swapability for easy servicing. 

The BIOS is easy to access and use, while the IPMI web interface provides a wealth of monitoring capabilities and easy remote management that matches the type of functionality available with Xeon platforms. Among many options, you can update the BIOS, use the KVM-over-LAN remote console, monitor power consumption, access health event logs, monitor and adjust fan speeds, and monitor the CPU, DIMM, and chipset temperatures and voltages. Supermicro's remote management suite is polished and easy to use, which stands in contrast to other platforms we've tested. 

Test Setup

Here we can see the selection of processors we've tested for this review, though we use the Xeon Platinum Gold 8280 as a stand-in for the less expensive Xeon Gold 6258R. These two chips are identical and provide the same level of performance, with the difference boiling down to the more expensive 8280 coming with support for quad-socket servers, while the Xeon Gold 6258R tops out at dual-socket support. 

To assess performance with a range of different potential configurations, we used the Supermicro 1024US-TR4 server with four different EPYC Rome configurations. We outfitted this server with 16x 32GB Samsung ECC DDR4-3200 memory modules, ensuring that both chips had all eight memory channels populated. 

We used a Dell/EMC PowerEdge R460 server to test the Xeon processors in our test group, giving us a good sense of performance with competing Intel systems. We equipped this server with 12x 32GB Sk hynix DDR4-2933 modules, again ensuring that each Xeon chip's six memory channels were populated. These configurations give the AMD-powered platform a memory capacity advantage, but come as an unavoidable side effect of the capabilities of each platform. As such, bear in mind that memory capacity disparities may impact the results below.  

We used the Phoronix Test Suite for testing. This automated test suite simplifies running complex benchmarks in the Linux environment. The test suite is maintained by Phoronix, and it installs all needed dependencies and the test library includes 450 benchmarks and 100 test suites (and counting). Phoronix also maintains openbenchmarking.org, which is an online repository for uploading test results into a centralized database. We used Ubuntu 20.04 LTS and the default Phoronix test configurations with the GCC compiler for all tests below. We also tested both platforms with all available security mitigations. 

Linux Kernel and LLVM Compilation Benchmarks

We used the 1023US-TR4 for testing with all of the EPYC processors in the chart, and here we see the expected scaling in the timed Linux kernel compile test with the AMD EPYC processors taking the lead over the Xeon chips at any given core count. The dual EPYC 7742 processors complete the benchmark, which builds the Linux kernel at default settings, in 21 seconds. The dual 24-core EPYC 7F72 configuration is impressive in its own right — it chewed through the test in 25 seconds, edging past the dual-processor Xeon 8280 platform. 

AMD's EPYC delivers even stronger performance in the timed LLVM compilation benchmark — the dual 16-core 7F72's even beat the dual 28-core 8280's. Performance scaling is somewhat muted between the flagship 64-core 7742 and the 24-core 7F72, largely due to the strength of the latter's much higher base and boost frequencies. That impressive performance comes at the cost of a 240W TDP rating, but the Supermicro server handles the increased thermal output easily. 

Molecular Dynamics and Parallel Compute Benchmarks

NAMD is a parallel molecular dynamics code designed to scale well with additional compute resources; it scales up to 500,000 cores and is one of the premier benchmarks used to quantify performance with simulation code. The EPYC processors are obviously well-suited for these types of highly-parallelized workloads due to their prodigious core counts, with the dual 7742 configuration completing the workload 28% faster than the dual Xeon 8280 setup. 

Stockfish is a chess engine designed for the utmost in scalability across increased core counts — it can scale up to 512 threads. Here we can see that this massively parallel code scales well with EPYC's leading core counts. But, as evidenced by the dual 24-core 7F72's effectively tying the 28-core Xeon 8280's, the benchmark also generally responds well to the EPYC processors. The dual 16-core 7F52 configuration also beat out both of the 16-core Intel comparables. Intel does pull off a win as the eight-core 6250 processors beat the 7F32's, though.  

We see similarly impressive performance in other molecular dynamics workloads, like the Gromacs water benchmark that simulates Newtonian equations of motion with hundreds of millions of particles and the NAS Parallel Benchmarks (NPB) suite. NPB characterizes Computational Fluid Dynamics (CFD) applications, and NASA designed it to measure performance from smaller CFD applications up to "embarrassingly parallel" operations. The BT.C test measures Block Tri-Diagonal solver performance, while the LU.C test measures performance with a lower-upper Gauss-Seidel solver. 

Regardless of the workload, the EPYC processors deliver a brutal level of performance in highly-parallelized applications, and the Supermicro server handled the heat output without issue. 

Rendering Benchmarks

Turning to more standard fare, provided you can keep the cores fed with data, most modern rendering applications also take full advantage of the compute resources. Given the well-known strengths of EPYC's core-heavy approach, it isn't surprising to see the 64-core EPYC 7742 processors carve out a commanding lead in the C-Ray and Blender benchmarks. Still, it is impressive to see the 7Fx2 models beat the competing Xeon processors with similar core counts nearly across the board. 

The performance picture changes somewhat with the Embree benchmarks, which test high-performance ray tracing libraries developed at Intel Labs. Naturally, the Xeon processors take the lead in the Asian Dragon renders, but the crown renders show that AMD's EPYC can offer leading performance even with code that is heavily optimized for Xeon processors. 

Encoding Benchmarks

Encoders tend to present a different type of challenge: As we can see with the VP9 libvpx benchmark, they often don't scale well with increased core counts. Instead, they often benefit from per-core performance and other factors, like cache capacity. 

However, newer encoders, like Intel's SVT-AV1, are designed to leverage multi-threading more fully to extract faster performance for live encoding/transcoding video applications. Again, we can see the impact of EPYC's increased core counts paired with its strong per-core performance as the EPYC 7742 and 7F72 post impressive wins. 

Python and Sysbench Benchmarks

The Pybench and Numpy benchmarks are used as a general litmus test of Python performance, and as we can see, these tests don't scale well with increased core counts. That allows the Xeon 6250, which has the highest boost frequency of the test pool at 4.5 GHz, to take the lead. 

Compression and Security

Compression workloads also come in many flavors. The 7-Zip (p7zip) benchmark exposes the heights of theoretical compression performance because it runs directly from main memory, allowing both memory throughput and core counts to impact performance heavily. As we can see, this benefits the EPYC 7742 tremendously, but it is noteworthy that the 28-core Xeon 8280 offers far more performance than the 24-core 7F72 if we normalize throughput based on core counts. In contrast, the gzip benchmark, which compresses two copies of the Linux 4.13 kernel source tree, responds well to speedy clock rates, giving the eight-core Xeon 6250 the lead due to its 4.5 GHz boost clock. 

The open-source OpenSSL toolkit uses SSL and TLS protocols to measure RSA 4096-bit performance. As we can see, this test favors the EPYC processors due to its parallelized nature, but offloading this type of workload to dedicated accelerators is becoming more common for environments with heavy requirements. 

SPEC CPU 2017 Estimated Scores

We used the GCC compiler and the default Phoronix test settings for these SPEC CPU 2017 test results. SPEC results are highly contested and can be impacted heavily with various compilers and flags, so we're sticking with a bog-standard configuration to provide as level of a playing field as possible. It's noteworthy that these results haven't been submitted to the SPEC committee for verification, so they aren't official. Instead, view the above tests as estimates, based on our testing.

The Xeon 6250 and 8280 processors take the lead in the single-threaded intrate tests, while the AMD EPYC processors post impressively-strong single-core measurements in the fprate tests. 

Conclusion

AMD has enjoyed a slow but steadily-increasing portion of the data center market, and much of its continued growth hinges on increasing adoption beyond hyperscale cloud providers to more standard enterprise applications. That requires a dual-pronged approach of not only offering a tangible performance advantage, particularly in workloads that are sensitive to per-core performance, but also having an ecosystem of fully-validated OEM platforms readily available on the market. 

The Supermicro 1023US-TR4 server slots into AMD's expanding constellation of OEM EPYC systems and also allows discerning customers to upgrade from the standard 7002 series processors to the high-frequency H- and F-series models as well. It also supports up to 8TB of ECC memory, which is an incredible amount of available capacity for memory-intensive workloads. Notably, the system comes with the PCIe 3.0 interface while the second-gen EPYC processors support PCIe 4.0, but this arrangement allows customers that don't plan to use PCIe 4.0 devices to procure systems at a lower price point. As one would imagine, Supermicro has other offerings that support the faster interface. 

Overall we found the platform to be robust, and out-of-the-box installation was simple with a tool-less rail kit and an easily-accessible IPMI interface that offers a cornucopia of management and monitoring capabilities. Our only minor complaints are that the front panel could use a few USB ports for easier physical connectivity. The addition of a faster embedded networking interface would also free up an additional PCIe slot. Naturally, higher-end Supermicro platforms come with these features. 

As seen throughout our testing, the Supermicro 1023US-TR4 server performed admirably and didn't suffer from any thermal throttling issues regardless of the EPYC processors we used, which is an important consideration. Overall, the Supermicro 1023US-TR4 server packs quite the punch in a small form factor that enables incredibly powerful and dense compute deployments in cloud, virtualization, and enterprise applications. 

MORE: Best CPUs

MORE: CPU Benchmarks and Hierarchy

MORE: All CPUs Content

We recently reported about Expanscape, a startup developing battlestation laptops featuring up to seven displays and offering their prototypes to interested parties at undisclosed prices. Today we can share some more information about pricing, which tops out at an eye-popping $20,692 (after conversion) for the seven-screen model's base configuration that's powered by an AMD Ryzen 5950X processor paired with Nvidia RTX 3070 (8GB) GPUs.

Expanscape's Aurora laptops with five or seven screens are a work in progress, so every unit is unique to a large degree. The manufacturer says that it is getting closer to finalized pricing for its A7 prototype as it had standardized on specific hardware, but until now, the company hasn't announced official prices of its multi-monitor laptops. 

Expanscape currently has two types of laptop prototypes in three configurations. The 'basic' Aurora A5 notebook comes with five monitors: four 15.6-inch 4K displays and one 7-inch touchscreen integrated into its palm rest. This system packs an eight-core AMD Ryzen 7 4800U processor paired with 64GB of DDR4 memory, a 2TB PCIe/NVMe SSD, and a 2TB SATA SSD. This entry-level machine currently costs approximately £4,500, or $6,286 USD after conversion. 

The 'full' Aurora A7 laptops are equipped with seven displays and come in two configurations powered by an AMD Ryzen 5950X processor paired with Nvidia RTX 3070 (8GB) GPUs. One model features four 17.3-inch 4K monitors, two 9.7-inch 1536p monitors, and one 7-inch touchscreen. Another does not have the 7-inch touchscreen but comes with a 17.3-inch touch-enabled LCD in its base to replace the keyboard, which now extends from under the chassis. 

Expanscape's Aurora A7 notebooks with seven screens are naturally more expensive than their smaller A5 brethren. At present, these machines cost £15,000 ($20,952 USD after conversion) for a standard model, but the pricing goes up with all of the customizations required for the professional built-to-order systems. All Aurora A7 machines come equipped with 128GB of DDR4 memory as well as 8TB of PCIe/NVMe and SATA storage. 

Both the A7 and A5 models come with internal 8 megapixel cameras with triple LED lighting and have dedicated hardware switches that can disable the cameras, microphone, speakers and wireless features. 

Notably, the current pricing of Expanscape's Aurora machines is somewhat higher than several months ago because of component shortages and new policies at the UK border. 

It is noteworthy that while Expanscape's Aurora notebooks are still prototypes, they are rather clumsy and heavy. However, according to the manufacturer, demand for these systems is still fairly significant. Customers who bought the systems reportedly said that they needed them 'yesterday.' 

What remains to be seen is whether high demand for Expanscape's Aurora A5 and Aurora A7 will enable the company to make them look and feel like commercial products and not just prototypes. Evidently, a more solid build will make these systems considerably more popular among interested customers. 

What if you need a mobile system with more than three screens? Well, Expanscape has developed a prototype of a laptop with as many as seven screens, and it is already selling the prototypes to interested customers. It also comes packed with an impressive amount of power on the compute side to match. 

The Aurora 7 Prototype indeed comes with four 17.3-inch monitors featuring a 4K resolution (two working in landscape, two in portrait mode) as well as three auxiliary 7-inch screens featuring a 1920 x 1200 resolution. All the monitors fold or swivel out of the primary chassis, making it a transformer of sorts, and no on-site assembly before deployment is necessary. The whole system weighs around 12 kilograms, so it is not easy to carry, but it is naturally easier to transport than a laptop along with six extra displays. 

With more of us working from home due to the pandemic, multi-display setups are becoming the norm and are widely used for a variety of applications. Setting a multi-monitor configuration at home or in an office is easy, and while attaching two more displays to a laptop is also possible, it gets slightly more complicated.

Expanscape's Aurora 7 Prototype laptop computer is built for very particular applications and audiences in mind (such as security operations centers, data scientists, content creators) that traditionally use multi-display PCs, but who at times need to transport and deploy them quickly. The creators wanted their seven-screen laptop computer to be portable, structurally rigid, and capable of running demanding programs. 

As far as internal hardware is concerned, the Aurora 7 is powered by Intel's Core i9-9900K processor that is accompanied by 64GB of DDR4-2666 memory, Nvidia's GeForce GTX 1060 graphics card, two PCIe 3.0 x4 M.2 SSDs, one 2.5-inch MLC SSD, and a 2TB 7200RPM hard drive. The PC has all modern connectivity technologies, including Bluetooth, GbE, Wi-Fi, and USB. Since the Aurora 7 uses a fairly spacious chassis, the developer says that it can use different platforms, including AMD's Ryzen 9 3950X or Intel's Core i9-10900K. 

Among the impressive peculiarities of the Expanscape's Aurora 7 Prototype are two internal batteries. The primary internal battery features an 82Wh capacity and powers the system itself. The secondary internal battery has a 148Wh capacity and is used to power the screens. The battery life for the whole system is about 2 hours 20 minutes, but at high clocks under high loads, it will get lower. 

Technically, all of Expanscape's seven-screen Aurora 7 machines are just prototypes that do not look or feel like commercial products, yet the company can build them to order and sell to interested parties who agree so sign a contract and pay a hefty sum of money. 

Supermicro has announced its first workstation based on AMD's Ryzen Threadripper Pro processor. The new system can pack a CPU with up to 64 cores, up to 2TB of RAM, and up to four double-wide graphics cards or accelerators. The machine is designed for AI/DL, digital content creation, and engineering simulations.  

Historically, AMD has been quite keen on winning designs with server makers, but when it comes to workstations from professional suppliers, the company has never been truly profound. Perhaps because high-end/corporate workstations is not exactly a high-volume business and spending resources on winning servers is always more fruitful. Yet, with its Ryzen Threadripper Pro 3000WX-series, AMD clearly wants to enter this market. The Supermicro A+ SuperWorkstation 5014A-TT is the industry's second workstation based on AMD's Ryzen Threadripper Pro 3000WX-series processor after Lenovo's ThinkStation P620 machine and it actually has an edge over its only rival as it supports more dual-slot PCIe 4.0 x16 cards, according to its manufacturer. 

The A+ SuperWorkstation 5014A-TT workstation is based on the AMD WRX80 platform with all of its advantages, such as an eight-channel memory subsystem that supports up to 2TB of DDR4-3200 SDRAM with or without ECC as well as 128 usable PCIe 4.0 lanes. The M12SWA-TF motherboard that powers the workstation carries six PCIe 4.0 x16 slots, four M.2-2280/22110 slots for SSDs, two U.2 sockets, one GbE port (Intel I210AT), one 10GbE connector (Marvell AQC113C), an ASpeed AST2600 baseboard management controller, a TPM 2.0 header, multiple USB 3.2 Gen 2 Type-A & Type-C ports, and a 7.1-audio codec.

The machine itself measures 535×222×573 mm and is equipped with an up to 2000W 80Plus Platinum-certified PSU, so it will handle virtually all combinations of AMD's Ryzen Threadripper Pro CPUs with any graphics cards, compute accelerators, SSDs, and memory modules available today. Meanwhile, since the A+ SuperWorkstation 5014A-TT is listed as 'coming soon', Supermicro does not disclose which add-in cards and storage devices it will offer with the system. 

The manufacturer has not yet disclosed pricing of its A+ SuperWorkstation 5014A-TT machines, but it is logical to expect its MSRP to be comparable to that of Lenovo's ThinkStation P620.

If you haven't heard of the gaming brand SuperO, we wouldn't blame you. Supermicro's gaming division isn't very well-known in the Western market. Nevertheless, SuperO's latest motherboard is pretty interesting. The Supermicro SuperO 9Z490-PGW is a high-end board with a total of four PCIe 3.0 slots, as spotted by Hermitage Akihabara. It is built on the new Z490 chipset with an LGA1200 socket, and, thus, for use with Intel's upcoming Comet Lake-S CPUs.

Typically, the Z490 platform doesn't support this many PCIe 3.0 slots. There are some boards that have this many slots, but few of them actually have all 16 lanes accessible on each slot. Supermicro's trick is to use a PLX lane switch chip, which essentially splits the PCIe lanes from the processor into multiple lanes. This overcomes lane restrictions put in place by graphics cards. Bandwidth per card is still limited.

Next to that unique feature, the board comes with all the standard goodies you would expect on a Z490 motherboard, including Intel i219V Gigabit Ethernet, Bluetooth 5.0, 802.11ax WiFi, four DDR4 RAM slots, dual M.2 slots and "enterprise quality" VRM circuitry. Iteven comes with 10 Gigabit AQC107 Ethernet built by Aquantia. A Realtek ALC1220 chip on a dedicated portion of the board handles audio. 

Intriguingly, despite its high-end positioning, it appears that the SuperMicro C9Z490-PGW doesn't come with RGB lighting. This paired with the PLX chip tells us that despite being labeled as a "gaming" motherboard, the new SuperO Z490 board is fit for a professional audience too. 

There's no word on pricing yet. However, PLX chips have become quite expensive in recent years, so don't expect this board to come cheap.

The era of 5G is coming, and it has the potential to shake up mobile internet connections in a big way. It can enable much faster data speeds for simple tasks from video chat to 4K livestreams, and it can even bring new capabilities to the table from Edge-powered augmented reality to smart infrastructure. But, the move from 4G to 5G isn’t as simple as flipping a switch. 5G networks and Edge computing require high-performance, open-architecture systems. To accommodate this transition, Supermicro is designing the backbone for 5G’s most advanced features. 

5G has a simple promise: deliver consumers high-speed data connections with massive bandwidth and low latency. 5G networks transmit vast amounts of information rapidly between devices and servers. Communications can go from one device to another in as little as a millisecond, and data transfers can hit speeds of up to 10 Gbps.

The improvements to speed and latency enhance the experiences already available on 4G LTE, but they also enable entirely new capabilities. Once 5G is commonplace, consumers will be able to stream video games to smartphones with high-quality graphics and negligible latency. Lightweight augmented reality devices will be able to offer a heavyweight experience. 5G networks can also have an impact on infrastructure, with more processing power to handle complex computing like traffic coordination.

Edge and cloud computing hardware is at the heart of all of those applications. Supermicro has various servers designed to fit the many needs along every point of 5G networks. The company has designed servers that are not just incredibly powerful but also highly efficient thanks to their inclusion of Intel® Select Solutions from 2nd Gen Intel® Xeon® Scalable processors to Intel® FPGA cards. Thanks to their efficiency, the servers have a low environmental impact and reduced energy consumption. Supermicro’s servers can also include leading hardware accelerator cards with GPUs for AI inferencing and FPGAs for advanced networking functions.

Supermicro has designed its servers to adapt quickly to the new needs of 5G, with smaller servers that can collectively form massive data centers. Supermicro’s Edge servers can even be built into rugged enclosures, increasing their lifetime in volatile environments. Supermicro has done just this with its customizable Outdoor Edge Systems.

These servers can withstand temperatures from -40℃ to 50℃, making them ideal for harsh environments. When the weather goes to extremes, a built-in, 300W heater and high-efficiency heat exchanger can keep the system up and running. The Outdoor Edge Systems are also protected against water and particles with an IP65 rating. Supermicro has further ensured they are GR-487-CORE and GR-3108-CORE compliant.

This all means that Supermicro’s servers can be deployed directly on 5G cell towers to keep data and processing power as close as possible to end users.

Supermicro’s Outdoor Edge Systems and SuperServers are powerful options for telecom and data center operators. Beyond their rugged design, they offer multiple PCI-E slots to enable GPU and FPGA acceleration or expanded storage capabilities. The promise of 5G is driven forward by Supermicro’s ability to put the hardware necessary for real-time AI inferencing, 5G RAN software, and much more right on the Edge.

As complex as these systems may all sound, Supermicro has refined the process of creating its 5G systems through its Building Block Solutions®. This enables simpler customization of each server to fit customers’ precise needs. So, where one system might be fully loaded with storage to deliver data as quickly as possible to end users, another might fill up the expansion slots with FPGA cards focused on a very specific workload. The 2nd Gen Intel® Xeon® Scalable processors can also flex to meet customers specific performance requirements.

Supermicro’s experience with open hardware and virtualization is uniquely suited to the needs of 5G. This ensures that every customer can find a solution that will flex to fit their needs. Supermicro already has a complete suite of server hardware available, but it’s continuing to work on even more 5G server infrastructure building blocks and solutions.

Its servers can bring storage closer than ever to users, delivering the fastest loading speeds. And, its hardware is ready to push AI forward through Deep Learning in data centers, increasing the capability of AI on its Edge servers. That AI could be tasked with everything from guiding self-driving cars to powering AR glasses, or even enabling advanced sensing devices for first responders.

If you want to prepare for 5G, you only have to look at what Supermicro is building. You can learn more about Supermicro’s 5G solutions here.

Update: As of Nov. 15, single units are still available only in UK.

The “more is better” philosophy of extreme computing has brought everything from CrossFire and SLI graphics arrays to workstation-class motherboards to the consumer PC market. So we were hardly surprised when companies started producing dual-motherboard cases (and even a desk) that lets users to install an ATX or EATX gaming platform on one side and a Mini-ITX number-cruncher or streaming PC on the other. But Supermicro has a thought for those of you who want real number crunching prowess on the Mini-ITX side: Why not go Xeon?

Born of a customer-exclusive design that Supermicro got permission to take public, the X11SRi-IF was originally intended to fit four complete configurations into a 2U chassis. For the consumer, that simply means that this board can support an LGA 2066 Xeon and four full-sized ECC registered DIMMs into the compact space meant to hold a Mini ITX consumer board.

Specifications

Features

 The first thing that makes this board frustrating from a consumer standpoint:  You’ll need a Xeon and ECC DRAM simply to use the X11SRi-IF, as the board won’t even start with an HEDT CPU such as the Core i9-7900X. And attempting to pair that Xeon with a non-ECC desktop UDIMM returns an error that only proves that these parts are electrically compatible, but not firmware supported.

Supermicro sent a CPU that would compete directly against our Core i9-7900X, the Xeon W-2155, along with four p/n MTA36ASF4G72PZ-2G6H1RQK sticks of Micron’s 32GB ECC registered DDR4-2666. While we can’t yet match the memory capacity with a consumer kit, the workstation processor runs the same 3.30 GHz base frequency and 4.50 GHz top boost frequency as the Core i7-7900X. Heck, the two CPUs even have the same 10-core, 20-thread configuration and 13.75 MB of L3 cache.

Another thing that isn’t consumer friendly (even for “prosumer” workstations) is an I/O panel that has only two USB3 Gen1 ports. This, on a board that doesn’t even have any internal USB headers. On the other hand, prosumers who are good at networking can do nearly everything that normal people do over USB via one of the two Gigabit Ethernet ports. We said two ethernet ports because the third RJ-45 connector is for remote system management via IPMI (Intel’s “Intelligent Platform Management Interface”).

The lower corner does support an old-fashioned serial port but, apart from the Power/Reset/HDD LED/Power LED header, there are no front panel connections. Not even audio, because the board doesn’t have a codec. And if you’re using this with a graphics card, the paucity of USB ports and lack of additional PCIe slots means it’s not getting audio, unless you use the graphics card’s HDMI audio pass-through. At least there’s that odd option.

What the Supermicro board does do better than most consumer boards of this size is connect three fans, rather than two. Three should be considered the minimum since most compact performance cases have a front, rear, and CPU fan, but we’ll understand if you’re willing to use splitter cables.

Lo and behold, this board also has a single 512MB DDR4-2400 integrated circuit (IC) serving its AST2500 Server Management Processor, an ARM11-based part that includes 2d integrated graphics. You’ll find a jumper on the bottom edge of the board to disable that feature.

While there’s no visible SATA, the metal-shielded header seen in the image above supports both PCIe x4 and 4-port SATA cables. Compatible data cables include Supermicro’s CBL-SAST-0933 four-drive SATA and its CBL-SAST-0956 NVMe (using SFF-8639 for U.2). Supermicro also takes that unusual step of providing SATA power via the two white four-pin connectors seen on the X11SRi-IF’s forward edge, each of which supports its CBL-SAST-0886 dual-drive breakout cable.

Expanding outward, we find an identical NVMe cable header between the CPU socket and I/O panel VGA port. This header is NVMe only, allowing the board to support two of the firm’s CBL-SAST-0956 data cables.

A third hit to the X11SRi-IF’s suitability for desktop users comes from its misplaced upper rear standoff, which was mandated by the board’s DIMM placement. We had to remove the standoff from our test system to prevent shorts, and our case didn’t have an alternative hole to match the X11SRi-IF’s unusual configuration.

You may have noticed that the X11SRi-IF has no 24-pin connector. But it does have a power button header: For solo use, an ATX-style power supply with EPS12V cable must be used with its power-on pin shorted to a ground pin, and power savings with the system off but the power supply on relies upon the power supply’s idle features. However, what seems off-putting to builders of compact PCs might be a blessing to those who build two systems within a single case, as some of those cases have only one power supply mount.

The EPS12V header feeds a 6+1 phase voltage regulator rated at 165W continuously, in addition to all onboard components including the SATA power cable headers. Once again, what sounds like a great configuration for a specific type of server can also be a perfect match to dual-motherboard systems.

The X11SRi-IF’s back side supports a single 2280 M.2 SSD upon a low-profile (4mm-tall) connector, similar to the back-side slots of other Mini-ITX boards.

The X11SRi-IF doesn’t support any overclocking features, so we tossed in a few dozen screenshots of the things its firmware does support, many of which are remote management features.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Benchmarks

Supermicro wants you to believe that its X11SRi-IF is unique in its support for a 140W LGA 2066 Xeon, but ASRock did the same thing for the Core i9 version of this CPU two years ago in its X299E-ITX/ac. Since the Xeon W-2155 and Core i9-7900X are supposed to be identical from the perspective of our benchmarks, this leaves the two boards to compete based on other parameters, which primarily relate to the type of memory used.

The X11SRi-IF requires ECC-registered DIMMs, and the ones we’re using have twice as many ranks and twice as many banks per rank as a standard 8GB UDIMM. While both features can dramatically improve memory performance, the same ECC technology that improves data accuracy can also put a damper on performance. Though we didn’t have a set of four non-ECC modules at the same capacity, we used a set of SODIMMs that defaulted to the same DDR4-2666 rating on the X299E-ITX/ac to keep things as fair as possible.

Sandra, 3DMark and PC Mark

We lead with memory performance just to see how badly ECC would impact its performance, or how much the extra ranks and banks would help. It’s obvious here that the performance advantage of doubling the ranks per channel and doubling the banks per rank far outweighs the performance deficit of verifying parity bits via ECC.

Looking at a few scores from 3DMark and PCMark, the Supermicro X11SRi-IF also appears to hold up its processor’s higher Turbo Boost ratios a little longer than the X299E-ITX/ac. Remember, both the Xeon and Core i9 have the same Turbo Boost rating.

3D Games

You probably won’t be using a Xeon for gaming, but if you do, you’ll be happy to see that the X11SRi-IF outperforms the X299E-ITX/ac in both Ashes and F1 2017. The later benchmark is known to be somewhat dependent on memory performance, particularly at its lower quality settings.

Mixed Workloads

The X11SRi-IF squeaks out a win in 7-Zip, but the X299E-ITX/ac solidly thumps it in Cinebench. The two appear evenly matched in our classic timed tests.

Power, Heat and Efficiency

The X22SRi-IF pulls much less power at idle than the X299E-ITX/ac, but the difference mostly goes away at full load. We had to test both with Prime95’s less-stressful “Large FFTs” because neither could hold both processors up to their rated Turbo Boost ratios when running the stressful Small FFTs test.

We should also note that our particular power supply can idle down to around 1W with the X299E-ITX/ac turned off, but the power supply can’t be turned off completely when connected to the X11SRi-IF and instead pulls up to 15W in standby mode. With different power supplies treating such a low-load condition differently, your results will vary.

The X11SRi-IF’s CPU ran a little cooler, but its voltage regulator a little warmer, using our Fractal Design Celsius S24 closed-loop cooler with fans atfull speed to provide the voltage regulator’s required airflow.

The X11SRi-IF performed a little more than 2% better on average than the X299E-ITX/ac, and based on our previous memory test we’re betting that most of that gain should be credited to the manufacturer’s choice of memory. We simply didn’t have a set of 8GB single-rank ECC DIMMs from which to produce a more closely-matched configuration.

Final Thoughts

Our performance-per-dollar chart is complicated not just by the fact that the X299E-ITX/ac has far more features or that the X11SRi-IF supports server processors and DRAM, but that Supermicro doesn’t yet have a US retailer for the board. Priced around £320 in UK tax included (and $330 US before tax), Supermicro is currently offering any distributor that wants these a low price of $256 in quantities of 2000. Thus, we used $300 as a best-estimate.

Pricing doesn’t get any easier for the X299E-ITX/ac, because this current model was always intended to be a limited release, and the sellers that still have these have tacked on a 50% surcharge for the privilege. We had to turn back to its original $400 MSRP simply to get back to basics.

The X299E-ITX/ac probably has $100 worth of extra features, and it supports a far cheaper processor model to provide the same number of cores and frequency. Yet for those who want ECC and advanced remote management, the X11SRi-IF’s feature set will appear superior.

In the end, it’s the lack of consumer-friendly I/O that has us considering the X11SRi-IF only as a prosumer number crunching platform, and not a primary PC. Server guys will be happy to hear that, but most end users at this point will probably just go with something AMD for a plethora of cores at a reasonable price.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

A famed hardware leaker known as @momomo_us on Twitter tweeted a diagram of what appears to be Supermicro's upcoming X12DPi-N motherboard as well as some interesting benchmark results of an unidentified 14-core, 28-thread Intel processor.

Intel's upcoming Whitley server platform for Cooper Lake-SP and, subsequently, Ice Lake-SP processors has been rumored to support the PCIe 4.0 interface for a while now.  

AMD has already adopted the PCIe 4.0 on its consumer Ryzen processors, so Intel is already late to the game. At this rate, it's hard to tell if PCIe 4.0 will arrive on an Intel consumer or server platform first. 

The motherboard diagram also shows the LGA4189 CPU socket, which TE Connectivity designs and produces for Intel. There are two variants to the LGA4189 socket, but we can't tell which version the X12DPi-N motherboard is using.

With two sockets, the Supermicro X12DPi-N was shown with a pair of mysterious Intel processors with 14 cores, 28 threads, 17.5MB of L2 cache and 21MB of L3 cache. The chip seemingly has a clock speed of 3 GHz and was working in tandem with 256GB of DDR4-3200 Micron-branded RAM.

It's uncertain if the 14-core chip belongs to the Cooper Lake-SP or Ice Lake-SP family (both support the LGA4189 socket). 

However, the L2 cache could be the key to solving the mystery. The processor seemingly sports 1.25MB per core, which doesn't match any configuration that we know (Skylake-SP and Cascade-SP Lake feature 1MB per core). Since Cooper Lake-SP is practically placeholder until Ice Lake-SP arrives, the chips will likely maintain the same L2 cache design. 

With the shift from Skylake to Ice Lake (14nm to 10nm), Intel increased the L2 cache from 256KB per core to 512KB per core. Now that the enterprise side is making this same 14nm to 10nm transition, there's a chance that Intel could upgrade the L2 again. Assuming the Cooper Lake-SP sticks to 1MB per core, then Ice Lake-SP arriving with 1.25MB per core seems possible. So if we had to guess, we'd say the 14-core, 28-thread chip is probably from the Ice Lake-SP family.

AMD held its European launch in Rome, Italy of its new EPYC processors. Along with announcing that the company's EPYC Rome data center processors have now attained 100 world records (and counting), the company also announced a slew of new OEM servers.

AMD also unveiled its EPYC 7H12, a 64-core 128-thread that boasts a beastly 280W thermal design power (TDP) envelope, which allows the processor to reach a 2.6 GHz base and 3.3 GHz max boost frequency, marking the highest performance of its Rome product stack. 

AMD designed the new chip, which requires watercooling to extract the ultimate performance, for high performance computing (HPC) workloads. ATOS unveiled its new Bullsequana XH2000, a hybrid supercomputer designed for exascale-class supercomputers that supports the new 7H12 chips.

The system supports the EPYC 7H12, cooling it's 32 1U blades (per rack) with an advanced water cooling solution, and said that it provides up to 4.2 TFLOPS of performance per chip, making the 7H12 up to 11% faster than the current top-of-the-stack EPYC 7742 processor.

As a quick refresher, EPYC's 7nm process brings density advantages over Intel's 14nm, which equates to higher core counts. It also brings power reductions (albeit with a 12nm I/O die caveat) that lead to more work done per watt (a critical consideration in the data center), higher clock speeds, more cache, and ultra-competitive pricing. Pair that with inherent cost and yield advantages of a chiplet-based design, a revamped Zen 2 architecture that brings an ~15% uplift to instructions per cycle (IPC) throughput, a fast move to PCIe 4.0, and an industry-leading serving of memory channels and throughput for x86 processors, and EPYC is no longer seen as the Intel "alternative." Now it's perceived as the leader in terms of leading-edge features that attract the heavyweights of the industry, as evidenced by the explosive uptake of Rome in the HPC and supercomputing space.

But general purpose servers see the lion's share of volume shipments, so having a full stack of enterprise-class systems, with support for the latest software and backed by OEM service contracts, is important to assure broad uptake. AMD has been on a full court press working with the leaders of the server industry to assure that it has a broad and capable stack.

Dell EMC recently announced its full lineup of enterprise-grade servers in both ultra-dense single-socket servers, which is a considerable focal point for AMD because its servers offer far more cores than Intel's Xeons in this form factor, and dual-socket models. Dell EMC also used its PowerEdge R7515 to set a new TPCX-V world record, notching a 280% performance improvement over its previous-generation EPYC server.

That record joins others posted by AMD's partners (not AMD itself), with the tally now reaching up to 100 world records for AMD's EPYC Rome processors. AMD also touted other solutions, such as systems from Supermicro that leverage EPYC's leading PCIe 4.0 connectivity by supporting a healthy ration of GPUs, and announced initiatives with IBM Cloud, Nokia, and OVHcloud, all of which join an ever-expanding ecosystem of Rome systems. AMD's foundry partner TSMC also made an appearance to discuss the company's close partnership with AMD and its joint R&D initiatives on future products. 

AMD also provided some comparative figures that provide a broad outline of EPYC Rome's value proposition relative to Intel's processors. As with all vendor-provided benchmarks, we should take these figures with a grain of salt, but there is no doubt that the EPYC Rome processors pressure Intel on multiple axes. Whip in broad vendor support with systems that feature enterprise-class support for both the hardware and software, and it's clear that EPYC Rome is emerging as a potent foe to Intel's Xeon lineup.

Researchers from security company Eclypsium have uncovered a set of USB vulnerabilities in in the baseboard management controller (BMC) on Supermicro’s server boards (models X9, X10 and X11) that could allow attackers to hijack thousands of servers. The researchers, the same ones who warned earlier that Supermicro's servers can be easily backdoored, named the vulnerabilities USBAnywhere.

Taking Over Supermicro’s Servers Remotely

BMC is a 'computer within a computer,' much like Intel’s often-criticized Management Engine (ME), that allows IT administrators to remotely control and update computers on a network. Normally, the BMC is locked within a network, so that it can’t be remotely accessed from outside of said network. However, not everyone does that, sometimes on purpose, if they would like to access some servers over a web interface. 

These interfaces aren’t typically designed with security in mind, either, which makes it that much easier for attackers to find existing vulnerabilities and exploit them. In this particular instance, the Eclypsium researchers found a vulnerability in the BMC feature that allowed IT admins to remotely mount images as USB devices. 

This could be exploited because it essentially gives hackers the ability to take over a device remotely as if they had local USB access to the machine. For instance, the attackers could install a new operating system or implant malware via the remote USB access.

BMC Vulnerabilities

The Eclypsium team found four vulnerabilities. These flaws included included using plaintext authentication and unauthenticated network traffic, weak encryption for the remote connection and an authentication bypass flaw in the Supermicro X10 and X11 platforms that would allow new clients on the virtual media service to use an old client's permissions. 

The Eclypsium researchers noted:

"If a valid administrator had used virtual media since the BMC was last powered off, the authentication bypass vulnerability would allow an attacker to connect even without the proper username and password.”

The security team that uncovered the bugs believes that tens of thousands of Supermicro servers could be open to this attack. The Eclypsium researchers said they contacted Supermicro, which has issued a fix for its server motherboards, but Supermicro’s customers will still need to install the update for protection against USBAnywhere attacks.

Enter for your chance to win the Supermicro Supero C9Z390-CGW Motherboard​.

Geared for the performance users who need the basics and the flexibility of all the latest technologies within a single motherboard, the SUPERO C9Z390-CGW appeals to the hardcore gamer. With gigabit wired and wireless networking, multi-channel audio, and RAID ready storage options, you will be ready to build your next gaming PC.

To enter head to the giveaway page and follow the instructions.

No purchase necessary. Void where prohibited. Giveaway is only open to legal residents of the fifty (50) United States, and the District of Columbia, 18 or older. For a complete list of rules please see the Terms and Conditions on the Gleam Giveaway Widget. This giveaway is open until September 6, 2019.

Hey there Tom's Hardware readers! From time to time we try to give back to the community with a giveaway. It's a win-win: You get free stuff, in this case the Supermicro C9Z3890 motherboard, and we get warm fuzzy feels. This mini-ITX motherboard has all the bells and whistles. It features the Intel Z390 chipset, supports up to 32GB of DDR4-3866 RAM, and has an M.2 Slot for all that SSD goodness. 

For your chance to win, follow the instructions in our giveaway thread on the Tom's Hardware forums. Good luck!

Last year, Bloomberg ran a report, saying Supermicro-supplied servers come with Chinese backdoors and that this may have been a reason for Apple to dropped them in 2016; although Apple denied espionage concerns at the time. Although new research publsihed today doesn’t exactly confirm Bloomberg’s report that Supermicro servers ship with pre-installed backdoors, it does point to the microcontrollers used by Supermicro and the firmware that comes with them being easily backdoored without detection. 

Researchers from Eclypsium, a firm specializing in firmware security, were able to commission a bare-metal server from IBM, install a backdoor in one of its microcontrollers, and then allowed IBM to re-use the server for other customers. The researchers were later able to reclaim that same server and noted that the backdoor was still active on the server, which means IBM lacks proper reclamation process that can clean previously used bare-metal servers of accidental or intentional backdoors. Attackers could use the same process that the researchers used to brick or steal data from other IBM customers.

Supermicro's “Parasitic Servers” Are Easily Exploitable

Previous research had shown that baseboard management controllers (BMCs), which are motherboard-attached microcontrollers, can give extraordinary remote access to servers inside data centers. The management capability on these BMCs is provided via the Intelligent Platform Management Interface (IPMI), which in many ways is similar to Intel’s Management Engine and its Active Management Technology and poses the same large risks of allowing attackers to take over servers remotely.

The IPMI is supposed to allow organizations to make configurations to a large number of servers remotely, even if the servers are turned off. In 2013, researchers pointed out that these BMCs can create “parasitic servers” that could allow attackers to take over entire server fleets in data centers.

Dan Farmer, a white hat security researcher, said at the time about BMCs and IPMI:

“Imagine trying to secure a computer with a small but powerful parasitic server on its motherboard; a bloodsucking leech that can't be turned off and has no documentation; you can't login, patch, or fix problems on it; server-based defensive, audit, or anti-malware software can't be used for protection; its design is secret, implementation old, and it can fully control the computer's hardware and software; and it shares passwords with a bunch of other important servers, stores them in clear text for attackers to access.”

Back then, the vulnerable firmware for these BMCs was developed by ATEN Technology, a Taiwanese company. These BMCs came pre-installed on servers from Dell, HP and other providers. A new paper is now showing that IBM’s servers are also highly vulnerable to undetectable backdoors installed in SuperMicro’s BMCs.

What Compromise of a Server's BMC Means

When an attacker compromises a BMC in a server, this opens up a variety of attack scenarios. For one, servers could be bricked, which means attackers could blackmail cloud service providers into paying them money “or else,” just as DDoS attackers have done in the past.

Having such low-level access to all the servers that are connected and controlled via a BMC means that the attackers can also extract all sorts of data from those servers from many cloud service customers. Ransomware and potentially crypto-mining malware could also be installed on many of the compromised servers.

The Eclypsium researchers recommended cloud service providers extend the reclamation process to the firmware level to ensure that any bare-metal reclaimed servers from a previous customer weren’t intentionally compromised.

The reclamation process should also include updating the BMC firmware manually, as well as the UEFI firmware via the BMC. The service providers should also continuously monitor changes to their servers’ firmware to spot attacks as they happen.

Over the past few years, more companies have come to realize that supply-chain security is just as important if not more important than applying software patches. Verifying that purchased hardware hasn’t been tampered with either from factory or somewhere in the supply chain should be an even bigger priority for cloud service providers who are responsible for the data protection of millions of customers.

Supermicro has reassured shareholders that a third-party investigation refuted a report from Bloomberg saying its motherboards had been compromised by Chinese intelligence agencies. The company said "the investigations firm found absolutely no evidence of malicious hardware on our motherboards," and with that being the case, it believes "today's announcement should lay to rest the unwarranted accusations" made about its products.

The claim that Supermicro's motherboards were compromised was published shortly after a Bloomberg Businessweek feature story entitled "The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies." Bloomberg said in its followup report that a telecommunications company security expert named Yossi Appleboum "provided documents, analysis and other evidence of the discovery" after "The Big Hack" was published.

Those reports were published on October 4 and 9. Supermicro CEO Charles Liang released a statement refuting Bloomberg's reporting on October 18, saying "no one has shown us a motherboard containing any unauthorized hardware chip" and explained that the company was "undertaking a complicated and time-consuming review to further address the article." The letter published today likely stems from that review.

Supermicro has not provided a lot of information about the review process, but that isn't uncommon among security companies. Most of the firms that investigate data breaches, hacks and other security problems specifically tell clients not to disclose their relationship to the public (which is why so many cite an "independent review," "third-party audit" or the like).

Bloomberg's "The Big Hack" and subsequent reporting immediately raised concerns throughout the U.S. after publication. But in the time since, many of the companies named in the report have denied its veracity. Apple CEO Tim Cook said his company wanted the story retracted, and Amazon publicly decried the report.

Bloomberg's news story about Supermicro's letter notes that "Bloomberg Businessweek has previously said that it stands by its story." At the time of writing, the publication has not made an any other statement about the situation.  

Features and Layout

It’s amazing to consider that a form factor as small as Mini ITX has found its way into the high-end enthusiast space. These small motherboards were originally built for simple tasks like running Internet kiosks or digital signs. However, today we have much smaller computers for those tasks and enthusiasts can competently cram a high-end consumer SSD onto a Mini ITX board with no problem. 

ASRock is the first company to properly power and cool Intel's Core i9-9900K in this compact factor with its Z390 Phantom Gaming-ITX/ac. Packed with features for such a small board, it's also surprisingly affordable. When we wrote this it was selling for about $190 (£145).

Specifications

Enthusiast motherboards these days are often overloaded with RGB. So the presence here of a mere three ground-effects-style RGB LEDs on the Phantom Gaming-ITX/ac’s underside is modest by those standards--but we're not complaining. Features include an oversized voltage regulator heat sink that’s designed to support real overclocking, dual M.2 slots and even a Thunderbolt 3 controller. All that said, keep in mind that this is still a somewhat value-priced product, so it only has a single Gigabit Ethernet port and a half-speed (20Gbps) Thunderbolt 3 controller. But that’s still more features than the limited on-board lighting and sub-$200 would suggest.

The Z390 Phantom Gaming-ITX/ac even kicks its Wi-Fi up to 1.73Gb/s, and does so inexpensively by using Intel’s new CNVi PHY to interface with the integrated controller of the Z390 chipset.

Aside from connections for the included Wi_Fi antennas, the back is loaded up with a CLR_CMOS button to the left, a PS/2 port for legacy peripherals, two USB 3.1 Gen1 ports, plus DisplayPort 1.2 and HDMI 2.0 for integrated graphics. The fun doesn’t stop there, as the Type-C connector located between four USB 3.1 Gen2 ports supports graphics output in addition to data (at 20 Gbps combined). Oh, and buyers still get Intel Gigabit Ethernet (I219V, to be precise), along with five analog audio jacks via Realtek’s high-end ALC1220 codec, and optical digital audio output.

ASRock has finally realized that even in the gaming PC market, Mini ITX buyers need (or at least want) two M.2 slots. The first M.2 slot is located inside the PCH heat sink, its heat spreader being semi-integrated into that sink. A heat pipe connects the chipset and voltage regulator sinks.

Builders can still use the M.2 slot located, by ASRock Mini ITX tradition, underneath the board. Both M.2 storage slots support PCIe 3.0 x4 and SATA interface drives, but while the top slot supports 60mm and 80mm drives, the bottom slot fits 80mm drives exclusively.

The Phantom Gaming-ITX/ac supports a slightly wider range of connections than the Z370 version it replaces, but it still has only 30 HSIO lanes and is still connected to the CPU by a four-lane DMI. That means, among other things, that leaning heavy on both M.2 drives at the same time will force them to compete for bandwidth. It also means that ASRock had to remove something to make way for the integrated USB 3.1 connections. The Phantom Gaming-ITX/ac has only four SATA ports (compared to the Z370 version’s six), and one of those is shared with an M.2 slot’s SATA interface. But the lack of a USB 3.1 Gen2 front-panel header will likely be a larger issue for builders who’ve picked a case that incorporates that feature.

Other documented headers include three 4-pin fan headers and one Addressable LED header on the top edge of the board. Also there live RGB LED, beep-code speaker, USB 3.0, an Intel-style front-panel button/LED headers, as well as a chassis intrusion header, along with front-panel audio and USB 2.0 headers at the back. Interestingly, there's an undocumented 8-pin header (in white) to the right of the board's 8-pin CPU power header, and another undocumented 9-pin header (in black) just behind the front-panel button/LED group.

The Z390 Phantom Gaming-ITX/ac accessory bundle is fairly light. In the box you'll find two SATA cables, an I/O shield, a Wi-Fi antenna, a driver disc, and documentation.

While the chipset’s limitations may make the use of a PCIe x2-based Thunderbolt 3 controller understandable, the seven-phase voltage regulator appears to be the Phantom Gaming-ITX/ac’s weakest specification. We’ll put that theory to the test later on in our overclocking evaluation.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Software and Firmware

Following the tradition it set with Fatal1ty F-Stream, ASRock’s A-Tuning gets yet a third name and color scheme with the Phantom Gaming Tuning app that's included with this board. It still bases its auto-tuning algorithm on existing overclocks, and the board lacks an EZ OC profile for our CPU. Therefore the app's Start button is inactive by default. Manual tuning from the board's OC Tweaker menu still works, however.

The Z390 Phantom Gaming-ITX/ac uses a 1.35V DRAM setting as default, even though our memory specified 1.20V for its DDR4-2400 default configuration. Phantom Gaming Tuning’s System Info page read a voltage of 1.376 to 1.384V, so we checked and found that 1.384V was closer to reality. In fact, we had to drop the firmware setting to 1.310V just to get a range of 1.348V to 1.352V, according to our voltmeter.

ASRock Polychrome RGB isn’t exactly configured to show the available lighting of this motherboard model, and instead wrongly labels the three LEDs on the underside of its bottom edge as being on the “PCH Heatsink.” The lighting controls work, but all three LEDs are on a single color channel, so rainbow effects aren’t available. The software is also compatible with our RGB memory, but clicking the “Sync” button drops color pattern options to “Breath”, “Blink”, and “Static” single-color modes.

Firmware

Like other ASRock boards, the Z390 Phantom Gaming-ITX/ac's UEFI defaults to its EZ Mode interface, uses the keyboard’s F7 function to switch into Advanced mode, and includes a setting under the Advanced mode Advanced menu to change the default GUI. While EZ Mode allows changing from default to XMP memory settings, boot order, fan profile, and even has a “Tools” menu for firmware update modes, you’ll need to switch to Advanced Mode to do anything as complex as disabling the onboard RGB lighting. Moreover, Advanced Mode includes ASRock’s OC Tweaker menu, with CPU, DRAM, and Voltage settings, in addition to five user profiles for which a set of custom CPU, DRAM, and Voltage settings can be stored. Oh, and if onboard storage somehow isn’t enough, users can transfer these profiles to a USB flash drive and share them with their friends.

Our overclocking begins with a CPU ratio of 49X at a BCLK of 101 MHz. That’s the same 4,949 MHz fixed clock as we’ve reached on other boards, and a 5% overclock from the Core i9-9900K CPU’s reference 4.70 GHz all-cores-loaded Turbo Boost ratio. Of course it’s still a bit less than the 5GHz available on up to two cores under light loads by default.

The Z390 Phantom Gaming-ITX/ac was the first board to give us hope for a 5.0 GHz full-load overclock using a mere 1.30V. It did manage to hold up for about 40 minutes before we saw a blue-screen error. Dropping to 4,998MHz by using 49 x 102 MHz just wasn’t enough of a decrease to get past the same (roughly 40 minute) crash time, despite several attempts.

“More voltage” is of course the trick to achieving overclocking stability, which is why Intel allows this CPU to reach up to 1.36V when using its two-core 5.0 GHz “Turbo Boost” setting. But keeping all eight cores cool at such high voltage levels is extremely difficult. We chose 1.30V to keep temperatures in check, and while our 240mm closed-loop cooler has a little extra capacity for higher voltage, we’re not changing our standards to accommodate one board.  Maintaining that voltage under changing loads was as easy as keeping the “Level 1” CPU Load-Line Calibration setting that the board automatically applied after we set the multiplier and core voltage.

Since there is no such thing as a perfect 1.350V, we’ve chosen a DRAM voltage limit of 1.355V for our memory overclocking comparison and prevent cheating via higher voltages by measuring the slots with a voltmeter. The Z390 Phantom Gaming-ITX/ac pushed an incredible 40mV over the set voltage. Worse still was that the extra voltage was obscured by a fictitious reading within the board's H/W Monitor report. The “1.308V” report shown above corresponds to a measured 1.348 to 1.352 volts at the slot while using the board’s 1.310V setting.

All that said, the Z390 Phantom Gaming-ITX/ac didn’t need the extra voltage to get a good data rate: Even after correcting its over-voltage, the board pushed our DDR4-2933 to DDR4-3818. But we’re actively trying to discourage a never-ending cycle of motherboard manufacturers sneaking increasingly greater voltage levels into their DIMM slots, only to have memory manufacturers respond with modules that need the extra voltage, prompting motherboard manufacturers to use even more voltage until the CPU’s integrated memory controllers eventually gives way at so-called “stock” settings.

ASRock’s Tool menu includes an outgoing email client for tech support messages, a tool for copying RAID drivers to a thumb drive, an SSD Secure erase tool, an Instant Flash firmware update mode, Internet Flash firmware download mode, and Network Configuration to help users reach the Internet Flash server prior to installing an OS. Internet Flash did reach the server this time and return a report of “no update available,” but a check of the motherboard’s webpage showed that an update was indeed available. After downloading the update manually on another PC, we used Instant Flash to install it, despite the Internet Flash feature insisting it didn't exist.

Only one of the three Z390 Phantom Gaming-ITX/ac fan headers provides voltage-based RPM control. Profiles can be manually configured by percentage or adjusted graphically under the FAN-Tastic Tuning sub-menu, and factory-programmed settings can be modified to match your specific fan’s profile using the Fan Tuning algorithm.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

How We Test

The second Mini ITX board we’ve tested using the Core i9-9900K, ASRock’s Z390 Phantom Gaming-ITX/ac goes up directly against Supermicro’s C9Z390-CG-IW. Full ATX models we included in our comparison give the tiny boards some big competition while helping to fill out the charts.

The same platform that cooled the 10 cores of our Core i9-7900X worked equally well with the eight cores of the Core i9-9900K, as we’ll show in the overclocking evaluation on the next page.

Comparison Products

Benchmark Settings

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Benchmark Results and Final Analysis

Most enthusiast-class motherboards employ a type of overclocking that ignores the CPU’s lower all-core Turbo Boost mode and instead deploys its highest single (or two) core Turbo Boost ratio regardless of the load. While most boards currently don’t do that with the Core i9-9900K, we still make sure the setting is disabled prior to our basic performance and power tests. We also enable all of the CPU’s power-saving features in firmware and leave the memory in its default (non-XMP) mode.

Synthetic Benchmarks

The two Mini ITX board here support only half of our four-DIMM RAM kit, and each module has one rank of 8GB. The Intel memory controller’s preference for at least four ranks of memory had us guessing about the performance impact of using only two ranks on the Z390-CG-IW. But things still look good in 3DMark for the Z390 Phantom Gaming-ITX/ac. It’s not that there’s no performance impact, but the impact is small.

Sandra Cryptography is bandwidth-restricted and impacted by the reduced ranks of the two-DIMM boards, as further illustrated in its Bandwidth benchmark. On the other hand, the performance gap we’d noticed on the competing Supermicro board went away for the Z390 Phatom Gaming-ITX/ac in Compubench.

3D Games

Now that we’ve seen it on a second two-DIMM board, we understand that giving up half of our memory ranks has some impact on all of our games. Though it affects a wider variety of games on the Core i9-9900K than we’d noticed in earlier 8700K tests, the difference appears smaller in the notoriously affected F1 2015.

Timed Applications

Handbrake completion times continue to rise even after we restored our benchmark image. We can only speculate that the SSD may be slowing after repeated reloads. Other than that triviality, the Z390 Phantom Gaming-ITX/ac performs nicely.

Power, Heat, & Efficiency

The Z390 Phantom Gaming-ITX/ac makes enormous strides in energy use, holding the 4.70 GHz eight-core Turbo Boost setting at a mere 1.16V and returning a 60W decrease in power consumption compared to the full-ATX Z390 ASRock Taichi. Conversely, the C9Z390-CG-IW appeared to use Intel’s 5.0 GHz two-core-boost voltage even when running an 8-core load at 4.70 GHz.

Lower voltage levels usually mean cooler temperatures, but we were still a little concerned about the Z390 Phantom Gaming-ITX/ac’s smaller voltage regulator and heat sink here. But it turns out there was no need for our concern.

Not only was the Z390 Phantom Gaming-ITX/ac the lowest-energy board in the mix, there was only a 0.8% performance deficit attributable to its use of only two modules from our four-DIMM kit. The result is an amazing efficiency score that's 18.7 percent above the average. And while buyers who are only willing to purchase two single-rank DIMMs could save a few watts on one of the larger boards, doing so would cancel out that 0.8-percent performance difference.

Overclocking

Everything we said at the top of the page about enabling power saving features goes out the window in overclocking. We’re seeking the highest 100-percent stable frequency while using Prime95 small-FFT’s to apply a 100% load across all eight cores and sixteen threads.

We mentioned in the firmware section that the Z390 Phantom Gaming-ITX/ac was the first board to run our CPU at 5.0 GHz for a full 40 minutes with all eight cores and 16 threads fully loaded, but that we had to give up on that setting after being unable to get it stable for much more than 40 minutes. We should also mention that the voltage regulator forced thermal throttling after around 20 minutes, and that we had to switch from a  roughly 60 CFM to a roughly 160 CFM fan to prevent that from happening. Dropping it back to 4.95 GHz was also necessary to get the voltage regulator temperatures under control. so this really is close to the limit of the stock voltage regulator cooler.

Memory overclocking was the only test we did on all of the big boards in both two and four DIMM configurations, and getting bandwidth numbers from Sandra is part of that process. This version of the bandwidth chart shows how each of these boards looks with two DIMMs at max O/C, and the Z390 Phantom Gaming-ITX/ac appears very competitive. Previous charts showed the 4-DIMM boards getting nearly 40GB/s when all the slots were filled.

Final Analysis

The Z390 Phantom Gaming-ITX/ac only needed to beat the other Mini ITX board Z390 board we've tested so far to declare a victory. It did so in performance, power consumption, heat, efficiency, and overclocking. Though most of the features between these boards are evenly matched, the ASRock board leads by having a Thunderbolt 3 controller--even if the controller only runs at half-bandwidth (20Gb/s). Of course, chipset pathway limitations meant something had to give, and the Z390 Phantom Gaming-ITX/ac lacks the front-panel USB 3.1 Gen2 header of the Supermicro C9Z390-CG-IW. On the other hand, if you have an older case, the ASRock board has the USB 2.0 header that Supermicro left out. Both boards have front-panel USB 3.0 headers.

Even if you never use Thunderbolt 3, it’s hard to deny the supremacy of a board that does everything else better than the competition while costing less. The Z390 Phantom Gaming-ITX/ac is even cheaper than the non-Wi-Fi version of MSI’s MGP Z390 Gaming Pro Carbon, though that won’t matter to anyone who needs the extra DIMM and PCIe slots afforded the ATX form factor. But for those in the market for a Mini ITX board to support Intel’s Core i9-9900K, the Z390 Phantom Gaming-ITX/ac is easily our first choice. 

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Features and Specifications

How do you solve a problem like that of the Intel Core i9-9900K's demands? Adding two extra cores to the Coffee Lake architecture meant it should automatically convert an extra 33 percent of power into heat. But Intel didn’t stop there. It also reverted to old-fashioned solder to join the CPU's core to its heat spreader, helping to eliminate the heat but doing nothing about the power requirement. And that worked so well that Intel decided to allow overclocking. With a core voltage limit that exceeds what we use to overclock, it’s hardly a shock that some boards come up a little short here. But before you write off the C9Z390-CG-IW, you’ll want to read the full story.

Supermicro C9Z390-CG-IW Specifications

Cramming a bunch of cores into a small space is nothing new, as we saw in our X99 and X299 Mini-ITX reviews, but the new Core i9-9900K is a different beast. First of all, hardly any enthusiast motherboards are power throttling it to its rated 3.6GHz to get it under its rated 130W thermal envelope under heavy loads. Second, most of the motherboards with an LGA 1151 socket that support it are based on previous designs meant only to handle the prior top-mainstream model, the Core i7-8700K.

We never got the opportunity to test the predecessor to Supermicro's new C9Z390-CG-IW, but our review sample's 7-phase voltage regulator certainly looks the part of a previous-generation mainstream board. Tiny sinks on those transistors (MOSFETs) do little to convince us of its Core i9-pushing power, but we’re hoping for a surprise larger than that of the 12-LED background RGB that spills from under the board’s leading edge.

Come to think of it, we already spotted one surprise in the board's M.2 storage slot, which fits neatly over the cutout section of its PCH sink. Most Z-series Mini-ITX boards have just one M.2 slot on the back, but the C9Z390-CG-IW has slots on both its back and front. And while both PCIe 3.0 x4 slots are designed to hold 80mm M.2 drives only, the rear one adds an SATA interface to support legacy drives.

The I/O panel includes a PS/2 port for legacy peripherals, two USB 3.1 Gen1 ports (aka USB 3.0), DisplayPort 1.2 and HDMI 2.0a, four USB 3.1 Gen 2 ports (including one Type-C), Gigabit Ethernet via Intel’s i219V PHY, 1.73Gb/s Wi-Fi via Intel’s 9560 CNVi PHY, six analog audio jacks and a digital optical audio output. A couple more USB ports would have been handy for power users, but we instead find a jumper in the panel’s only vacant space.

The jumper allows the board to force the PCIe x16 slot to function in x8/x8 mode for use with double-slot riser cards, which sounds incredibly practical coming from a company that cut its teeth on server parts. Making the switch still requires that the CPU support PCIe bifurcation, but that feature is common to all Intel's Core i9, i7 and i5 CPUs that use the LGA 1151 socket.

A few of the C9Z390-CG-IW’s headers are designated for factory use or undocumented, but there are USB 3.1 and 3.0 front headers located above and before the top-side M.2 storage interface, along with front-panel audio in front of the I/O audio jacks, a TPM header in front of the PCIe x16 slot latch, as well as an RGB strip and two fan headers on the top edge. Oddities include an OC mode jumper under the PCIe x16 slot’s lower edge, an SATA Disk-On-Module power header in front of the TPM header and a Supermicro-specific front-panel LED/button section between the two fan connectors that includes an output for an overheat/fan failure LED.

The C9Z390-CG-IW includes a printed manual, driver disc, I/O shield, cable label stickers, two Wi-Fi antennae and two SATA cables.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Firmware

Recent years of developing a consumer-friendly UEFI has resulted in an EZ Mode GUI from Supermicro, along with advanced settings available via the keyboard’s F7 function. Users unfamiliar with overclocking can still set boot order and enable XMP memory profiles from EZ Mode. Charging straight into Advanced mode, we found an Overclocking menu that presented CPU, Memory, Graphics and Voltage submenus.

The CPU Overclocking submenu includes basic overclocking configurations for:

• 4.3 through 4.6GHz using a 1.3V CPU core, 1.3V System Agent and 0.95V CPU I/O
• 4.7 and 4.8GHz using 1.35V CPU core, 1.30V System Agent and 1.2V CPU I/O
• 4.9GHz at 1.4V CPU core, 1.35V System Agent, 1.2V CPU I/O and Level 4 Load Line Calibration
• 5.0 through 5.5GHz using 1.45V CPU core, 1.4V System Agent, 1.2V CPU I/O and Level 4 Load Line Calibration

Since our CPU overheats at 1.35V, we started with the 4.6GHz setting and worked our way up to 4.94GHz at 49 x 101 MHz.

Default power limits were insanely low, and even overclocking modes don’t appear to be designed with an 8-core, 16-thread CPU in mind. We increased both CPU power limits (second image above) to the motherboard’s limit to prevent overcurrent protection from resetting the board.

The DRAM conundrum should feel familiar to old-time overclockers: Setting XMP profile blocks out individual timings and data rate. Switching from Default to Custom memory profile allows users to tweak everything from Default baselines, and switching from XMP to Custom memory profile allows users to tweak everything from XMP baselines. Unfortunately, users who don’t know what to set for all 66 adjustments aren’t likely to get very far, as only seven of those can be configured to automatically adjust based on a manually selected data rate.

The Load Line Calibration feature adjusts voltage upward in response to it dropping under heavy software loads. We found that the easiest way to get 1.3V CPU core when using 16 threads of Prime95 (small FFTs) was to choose a 1.285V core setting and Level 2 Load Line Calibration. Concerning Load Line Calibration, Level 1 put our core voltage through the roof, while Level 3 allowed it to drop well below our desired operating level.

The non-overclocking menus for CPU and Memory settings allow users to make non-overclocked adjustments. We enabled all of the CPU’s power-saving features for comparative benchmarks and power measurements but not for overclocking evaluation.

RGB control is found within the Boot Feature submenu of the Advanced menu. However, "on” and “off” are its only modes, and the only “on” profile is a fast-flashing rainbow mode.

The C9Z390-CG-IW has only two fan headers, and both of those support PWM-based settings only. Automatic modes appear to control a three-pin RDM1225S, despite the PWM-based settings of manual profiles.

The Save & Exit menu includes everything from boot settings to two registers where users can save their firmware configuration as profiles. The board defaults to AHCI storage mode and must be manually configured to access UEFI-enhanced OS installations.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

How We Test

The C9Z390-CG-IW is our first Mini-ITX motherboard designed to support Intel’s new 8-core, LGA 1151 processors, so we have only ATX boards to use for comparison. The ASRock Z390 Taichi, MSI MPG Z390 Gaming Pro Carbon and Gigabyte Z390 Aorus Master fill those roles.

The same platform that cooled the 10 cores of our Core i9-7900X worked equally well with the eight cores of the Core i9-9900K, as we’ll show in the overclocking evaluation on the next page.

Comparison Products

Benchmark Settings

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

Benchmark Results and Final Analysis

We enabled all CPU-based power saving features in firmware and disabled fixed-mode Intel Turbo Boost manipulations prior to our basic performance and power tests. The maximum Turbo Boost ratio for the Core i9-9900K is 5GHz with up to two cores loaded, and that drops to 4.7GHz when more than four cores are loaded.

Synthetic Benchmarks

While some boards enhance advanced memory timings to get a small advantage in a few benchmarks, the C9Z390-CG-IW suffers from the fact that the boards its being compared to have four DIMM slots. Given that Intel processors consistently show a performance preference for a minimum of four memory ranks and that each module of our four-DIMM kit is single-rank, we’ll cut it a small break.

The C9Z390-CG-IW started out behind the four-DIMM boards at 3DMark’s lowest effort test but gradually caught up as the workload increased. Sandra Memory Bandwidth shows the deficit was due not to a problem with the board, but the CPU’s preference for more memory ranks. The mix put it on par with the others in Cinebench but a stroke or two behind the pack in Compubench.

3D Games

Gaming benchmark history would have us believe that the two-DIMM C9Z390-CG-IW should fall farthest behind at medium settings in F1 2015 and take a minor hit in Ashes of the Singularity, but that’s not exactly what happened. The “good enough” score in Ashes was a little farther off than expected, and the big hit in F1 2015 came at its ultra preset. Perhaps the newer CPU is pushing us to the limits of the graphics card and DRAM?

The oddest result was in Metro Last Light Redux, which our previous configuration showed only took a performance hit when saddled with memory slower than the baseline of these modules. Our expectations are slowly evolving to accommodate the newer CPU.

Timed Applications

Where less time means more performance, the C9Z390-CG-IW keeps us very close to schedule. However, we did see a 30-millisecond loss in PowerPoint.

Power & Heat

This is where things get weird. Sometimes, the C9Z390-CG-IW would shut down when we started Prime95 small-FFTs. Other times, it would shoot up to around 242W before permanently throttling to 3.4GHz, well below the CPU’s rated frequency. Setting a higher power limit in firmware occasionally allowed us to reach 4.7GHz, where we eventually found a 344W full-load power reading. Most of the time, the higher power limit would push the voltage regulator to its critical limit, causing it to power off. It wasn’t until we turned Intel Hyper-threading off that the voltage regulator stayed at full load long enough to get a proper VR thermal reading. Even then the CPU was only pulling 297W.

A default voltage range of 1.32 to 1.36V for the CPU core was the only data we could find to explain the power throttling issue and high wattage and CPU temperature. In fact, the CPU temperature crossed the board’s default threshold of 100 degrees Celsius (212 degrees Fahrenheit), which resulted from the motherboard’s default setting of 125 degrees Celsius (257 degrees Fahrenheit) being out of range for the CPU. Since the CPU’s thermal limit is 115 degrees Celsius (239 degrees Fahrenheit), restricting the motherboard via a 10-degree thermal offset actually enabled the 100 to 115-degree range.

The great news is that performance was down by only 1.9 percent, despite the performance deficit our CPU encounters when switching from four to two ranks of memory. The C9Z390-CG-IW performed more competitively than we would have expected from a two-DIMM board.

Overclocking

By now you’re probably wondering “If this board was tripping its own circuit breaker at stock CPU settings with power-throttling disabled, how can it be overclocked?” The board does have an OC-mode jumper, but enabling that boosts CPU core voltage to 1.48V, tripping the board's circuit breaker all the faster. But you may also remember that we said it was running up to 1.36V CPU core at stock settings. We actually use less voltage to achieve our overclock!

At 1.3V CPU core, the C9Z390-CG-IW will run all day at 49 x 101 MHz under Prime95 small-FFTs. That’s 16 super-AVX-loaded threads! Could this be the first board we recommend exclusively for overclocking and never for use at the board's default i9-9900K settings?

Even though the C9Z390-CG-IW didn’t make the necessary adjustments to secondary and tertiary settings to properly overclock our DDR4-2933, it did let us run those settings at the same 101MHz base clock. (If you've noticed that the charted bandwidth advantage of the four-DIMM boards is reduced from previous reviews, that's because we’re showing their two-DIMM results.)

Final Thoughts

Our C9Z390-CG-IW review was a wild ride, primarily because the board's default settings couldn’t run our CPU at its full 4.7GHz 8C/16T rating under Prime95. The board throttled our CPU below its non-turbo-boosted 3.6GHz at stock settings and often tripped its thermal protection mechanism with power throttling disabled.

The C9Z390-CG-IW still looks good in a performance-per-dollar chart, and it’s the first Z-series Mini-ITX board we’ve seen in a while with two M.2 storage slots. The biggest problem is that we really can’t recommend it to anyone who wants to run Prime95 small-FFTs on the Core i9-9900K. That may not be a big deal to you, but our reviews require consistency.

Our findings around disabling Hyper-threading (see Power & Heat, above) is especially relevant for those with a Core i7-9700K, since that CPU doesn’t have Hyper-threading. The C9Z390-CG-IW had no trouble running our 9900K with Hyper-threading disabled, and the 9700K presents an even lighter load by being at least 100MHz slower.

For i9-9900K ovecrlocking, you could just overclock at a reasonable voltage, such as the 1.3V we chose. The 4.94GHz we reached under an 8C/16T load still beats the stock 4.7GHz by 5 percent. And if you care enough, we did try setting a -50mV offset at stock frequency settings and came up with a 1.22V CPU core drawing 256W at 4.7GHz in Prime95 small-FFTs. The fix is there, but applying it is up to the buyer.

MORE: Best Motherboards

MORE: How To Choose A Motherboard

MORE: All Motherboard Content

From Duplo Dragons to Giant Chassis

We saw plenty of award-worthy products here in Taipei and filed dozens of news posts from the show floor. But just like at any big trade show, there were several eye-catching, sometimes quirky things that fall outside the bounds of our usual coverage, because maybe they aren’t quite brand-new, they aren’t strictly products you will be able to buy, or they fall a bit outside of our usual coverage areas.

They're still well worth highlighting, so we’re recognizing them here where we’ve picked the 21 coolest things we saw (and didn't give awards to) at Computex 2018.

Tesoro Gram XS

If Dave Bowman from 2001 was shopping for a mechanical keyboard, he'd get Tesoro's futuristic Gram XS. The keyboard has a great stark-white aesthetic with flat keys and a super-slim profile that make it look like it belongs on Discovery One. Vibrant, customizable RGB lights shine through the flat, chiclet-style keys.

Despite its slim dimensions, the keyboard felt pretty good when I ran briefly tried typing on a model with the company's own clicky blue switches. If you like a quieter keyboard, you can get it with red switches. You can even buy it in black -- the same boring color as most keyboards -- but I'm sorry Dave, I'm afraid I can't do that. The Gram XS already on the market and available for $119. -- Avram Piltch

InWin Z-Tower

Case maker InWin is known for showing off crazy concept chassis at trade shows—and often bringing them to market. Recently, they’ve been high-tech and motorized, looking like something from a Sci-Fi effects shop. But the company’s latest, the Z-Tower, is more akin to a work of modern art.

Made of nearly 90 pounds of cast aluminum in a sweeping, almost organic-looking spiral and standing several feet tall, it was the one case at Computex 2018 that was impossible to ignore. If you think you’d like to build your next PC inside the Z-Tower, be prepared to make lots of room—in your home and your bank account. The company says the Z-Tower will likely sell for several thousand dollars. -- Matt Safford

Lian Li Desk PC (with Dual Computers)

Meet the ultimate piece of geek furniture. Lian Li has been making desks that double as PC cases for a few years, but this year's show was the first time I got to see them up close. The company showed off two upcoming models, one that holds a single computer and another with dual chambers so you can have one PC that plays games and another that streams (or a server) in the same desk. Both models have hydraulic lifts that you can adjust to turn this into a standing (or kneeling on the floor) desk.

There are USB ports and a power button on front lip. A clear glass surface lets you see all the glory of your components at all times. Most importantly, the entire top is free for your monitors, keyboard, mouse and other stuff, with no space devoted to a tower case. -- Avram Piltch

Cougar Minos One

Any time you add a screen to a device that doesn't normally have one, that's a win. Cougar's Minos One is one of two mice with color displays that I saw at Computex, with the other being Coolermaster's MM830.While the Coolermaster's pointing device has a great feel, the Minos One has a much larger panel on board.

When I saw the Minos One on the show floor, its screen was showing an animated Cougar logo, but a Cougar rep told us that, when this product launches later this year, it will be able to show real-time information like the DPI setting or in-game statistics. This wired mouse also has two custom RGB lighting zones and the ability to run at up to 4,000 dpi. -- Avram Piltch

Corsair Obsidian 1000D Case

In 2018, the hot question is no longer "how many Cores does your computer have" but "how many computers does your computer have?" There are other chassis on the market that hold two discrete systems inside, but Corsair's Obsidian 1000D is one of the biggest.

At 27 inches tall, 27 inches deep and a full 12 inches thick, the Obsidian 1000D is large enough to make your enemies quake with your fear and your frenemies light their RGBs green with envy. Ideal for gaming on one motherboard while streaming on the other, the case has room for 18 different fans and four 480mm radiators. It also provides space for up to five 3.5-inch hard drives and up to six 2.5-inch SSDs.

You can buy the Obsidian 1000D now for $499. -- Avram Piltch

Lian Li Strim RGB Cable

If your fans have RGB lights, your power supply has RGB lights, your RAM has RGB lights and even your power supply has RGB lights, why not your cables? Lian Li's upcoming Strim is a customizable RGB cable that runs from your power supply to your motherboard.

There's also a version that goes from your PSU to your graphics card. If you're going to bling out your system, you've gotta go all the way. -- Avram Piltch

Gamdias Hades P1 RGB and Nyx P2 RGB

Despite the inconvenience, most hardcore gamers prefer the accuracy and low-latency of a wired mouse to a wireless one. However, with Gamdias's Hades P1 RGB, you get the best of both worlds; a 12,000 dpi wireless mouse when you want it and, if you plug in its USB cable, a direct wired connection to your PC. More importantly, this attractive pointing device has customizable RGB lighting and a choice of side panels you can pop on or off to change your grip.

The Hades charges wirelessly using Qi technology and that's where Gamdias's other really-cool upcoming product fits in. The Nyx P2 RGB mouse pad not only has a Qi charging pad built into its upper left corner but also features an attractive ring of customizable RGB lights. Both products are due out later this year. -- Avram Piltch

Thermaltake H200 RGB Headset

We didn't have a headset award category on Tom's Hardware this year, but based on my brief experience with Thermaltake's H200, I'd award the company $40, the MSRP, of my own money to buy one. On the outside of the over-the-ear pieces, you'll find customizable 3D RGB lights that really pop.

Meanwhile, the padded ear cans felt extremely comfortable when pressing against my ears. They seemed to be just the right size: not so small they have to go in your ear and not so large that they smush your lobes. I didn't get to listen to the output, but my colleague, Sherri Smith heard some rich sound coming from the device. For me, the comfort, looks and reasonable price puts this device on my personal wish list -- Avram Piltch.

Patriot Viper V765 Keyboard

Gamers want switches that go up and down quickly so they can shoot or move faster, but many typists, including yours truly, can't live without clicky keys. Patriot's upcoming Viper V765 is one of the first keyboards to use Kaihl Box White switches, which have a lower actuation point than blue switches, but still have that snappy feel and pleasant click sound.

When I tapped the keys on the Viper V765, they felt like they could offer the perfect balance between speed and tactile feel. Typing comfort matters most, but this keyboard also has great looks, thanks to its aluminum chassis and fully-customizable RGB backlighting. It even comes with a comfy wrist rest that attaches or detaches magnetically. -- Avram Piltch

Corsair iCue Software

Every company that makes RGB peripherals or components has its own control software. However, Corsair's new iCue application takes lighting customization to the next level. iCue gives you incredibly fine grain control over each LED in your system;it lets you create different effects for every key on your keyboard or even every single light inside every single one of your case fans.

Better still, iCue can change the light show based on what happens in the game you're playing. It was really cool to see the demo system's case, keyboard and mouse all flash yellow when the main character caught fire in Far Cry 5. So far, Far Cry 5 is the only game that works with iCue, but hopefully more are coming. You can also set your lights to react to your system temperature, so they show that your CPU is "red hot." -- Avram Piltch

ASRock Optane 905P M.2 RAID Array

RAID 0. When you absolutely, positively have to get ridiculous transfers speeds, accept no substitutes. Now, what happens when you take a bunch of Intel's blindingly-fast 905P M.2 SSDs and put them into a RAID array? ASRock decided to find out by connecting four of the drives and connecting them together using a VROC card.

The results are truly impressive, with a sequential read rate of 10.7GB per second and  9.7GB per second of write speed. True, the 4K random reads were worse for the RAID array than for a configuration that has just one disk. Just don't put this array anywhere near your Thunderbolt port, because it's so fast that it may take you on a journey back to 1955. -- Avram Piltch

The Inception Project (RGB PC Wall)

You can think of it as everything that’s wrong with modern PC trends, or PC art on a massive scale. But it was impossible not to stop and stare at this mesmerizing wall of liquid-cooled, RGB-lit PCs on display at the Nangang Convention Center. I can say this with certainly because nearly every stick of RAM, AIO cooler, and chassis I saw at Computex sported its own similar rainbow light show.

Yet, at least twice during the long week of endless meetings and writing, I found myself standing mesmerized in front of Inception Project. Maybe it was the scale of the whole thing, or maybe my brain just needed a break from the madness of the show. -- Matt Safford

Ducky Year of the Dog Keyboard

There were plenty of interesting new keyboards at Computex. But Ducky’s limited-edition Year of the Dog mechanical clacker managed to catch my eye from a couple booths away without the use of bright RGB LED lights. (They’re there embedded in the Cherry MX keys, but they weren’t lit when I was ogling the keyboard).

Designed with the help of Hong Kong tattoo artist Michael Chan, and covered with art based on Chinese mythology, this may be the first keyboard that deserves to live in an art gallery. That said, if you’d like to take one home you’d better be ready to order come August or September. Ducky says that’s when the Year of the Dog Keyboard will go on sale, but only 2,018 will make their way off the assembly line. I’m sure fans of Asian gangster cinema--or just fans of tattoo art--will gobble them up quick.  -- Matt Safford

Well Buying Industrial’s Briefcases Full of LED Switches

The corners, crevices, and back walls of trade shows are often the best places to find niche products that are intriguing--or just plain odd. Well Buying Industrial (Co., Ltd) had a small booth full of all kinds of LED switches. Now, there’s nothing all that strange about switches. But the presentation, with dozens of models all lit up inside a collection of briefcases, made it seem as if they were going to be sold door-to-door. -- Matt Safford

MSI’s Massive Duplo Lucky Dragon

No trade show is complete without spotting Lucky, MSI’s dragon mascot. The company has taken to bringing a huge stuffed Lucky to CES.

But this massive model at MSI’s headquarters, made from Duplo bricks and standing at least 10-feet tall, was a truly pleasant surprise. It’s tough not to smile when you bump into a room-sized plastic cartoon dragon. I’m just glad I didn’t have to help put Lucky together. -- Matt Safford

Noctua Desk Fan Prototype

Noctua makes some of the best fans in the PC business, thanks in large part to serious engineering and research into the finer points of airflow. With this prototype, the company is taking its expertise outside of the PC realm and onto the desk, in a serious endeavor to keep you cool during those hot summer days.

Using the company’s newly redesigned NF-A12x25 fan, plus a cone designed to maximize and adjust the airflow passing through the fan, the device delivered a surprisingly strong breeze from such a small device. It was just what I needed after stepping out of Taipei’s sticky humidity and onto the show floor. A Noctua rep told me this desk fan will very likely make it to market, though there will be some aesthetic and design changes, along with a USB power cable so you can run the fan from your PC or a power bank. -- Matt Safford

Noblechairs Hero

This year, Computex looked like it was half tech expo, half furniture store, because so many different vendors showed off their new gaming chairs. Noblechairs upcoming Hero model stood out from the crowd by focusing on comfort, ergonomics and a refined aesthetic.

Unlike most companies, which use cheaper components like wood or bonded leather in their models, all of Noble's chairs have a robot-generated metal frame and your choice of durable materials such as polyurethane or real leather.

Due out within the next few months for an estimated starting price around $450, the Hero is the executive chair I want for my office. Forget about play time, because this chair is just as good for working long hours editing a report as it is for all-night gaming sessions. It features built-in, adjustable lumbar support rather than a pillow and everything about the chair feels premium, from the tautness of the material to the tight hinges and dials you use to adjust it.

I had a chance to go butts-on with the Hero at Computex and, rather than leaning back, it made me want to roll up to a desk and start typing. This chair could actually make you more productive. -- Avram Piltch

MSI Gaming Premium Stack (GPS)

Headset stands are the next frontier for innovation. At Computex, I saw several different stands with USB hubs in them, RGB lights and even surround-sound speakers. However, MSI's Gaming Premium Stack, aka GPS, stands out from the crowd, because it has a holographic screen on it.

When it comes out later this year for an estimated starting price of $399, the Stack will feature a stand, an optional wireless charger and the Lucky Box, which is a holographic screen that shows animations or system information floating in a 3D glass tube. MSI showed us the control software which lets you choose between a couple of default animations of company mascot Lucky the Dragon or show stats such as your computer's system temperature. You can also upload your own animated GIFs. -- Avram Piltch

ASRock Superb Mining Rig with Seven Phantom RX 580 Cards

ASRock’s recently launched Polaris graphics cards are so new, I’d never actually seen one in person before the show. And yet, here sit seven Phantom RX 580s tightly packed into an “ASRock Superb Mining” rig. Perhaps the only thing more superb than the hash rate of a rig like this: the money it could have made mining, had ASRock found a way to produce these cards in 2017, before the current coin mining slump. -- Paul Alcorn

Noctua Chromax Heatsink Cover Mod

I’ve seen all sorts of case and component mods over the last couple of decades, first as an enthusiast, then as a journalist. But these painted heatsink covers for the Noctua Chromax NH-D15 cooler line really stood out on a show floor dominated by glass and RGB. Created by Portuguese graffiti artist Le Funky, these covers show off the possibilities of creating a truly unique heatsink setup if you decide to trick out the heatsink covers the company sells for a few of its coolers.

Now, I’m not saying a custom-painted cooling cover would be beneficial for your CPU temps. But if you’re after some standout aesthetics for your next build and you have an artistic streak, it’s probably worth trading a few extra degrees of load temperature for a cooling tower that’s truly unique. -- Matt Safford

Supermicro Dual-Socket Gaming System

This week, we heard a lot about an exotic server processor and platform repurposed for the desktop PC, but at least Supermicro is straightforward about it. Supermicro's new dual-socket gaming system is designed for the 0.05% that want the absolute most power they can get for a gaming system.

This dual-socket system supports two Xeon Scalable processors of your choice, but you'll need one with high clock speeds to push this gaming system along. For the most well-healed, you could slap in two Xeon Platinum 8180's, which weigh in at $10,000 apiece, and create a  56-core 112-thread monster. Supermicro also touts ECC support as a feature, but I'm not sure how that's relevant to gaming, as it certainly won't give you a boost. In either case, I'd sure like to take this rig home, but I'm coming up about $20,000 short. -- Paul Alcorn

LAS VEGAS, NV -- Supermicro had its new flagship C9X299-PG300 motherboard on display in its suite at CES 2018. Notably, the company specifically listed support for Intel Core X-Series CPUs with up to a 300W TDP. Currently, the 165W TDP Core i9-7980XE has the highest TDP rating of the X-Series, so the 300W rating is an interesting wrinkle. We asked Supermicro for more details, but the company said it couldn't share any more information. 

Supermicro may not be the first name you think of when it comes to gaming motherboards, but the company is looking to change that. Supermicro knows, much like the other vendors, that capturing the enthusiast's eye, and their build, requires a strong focus on aesthetics in addition to the normal overclocking bells and whistles. Striking that balance isn't easy, and the company previously tried a less-conventional route with a bright green and black color scheme. It has since discarded that scheme in favor of the black and grey motif seen on the C9X299-PG300.

The X299 motherboard comes in the standard ATX form factor and features eight DDR4 slots that Supermicro says supports 4,000MHz+ after tuning. You also get six SATA 3.0 ports and two U.2 ports, along with two M.2 slots hidden under the long black extensions between the PCIe slots. These beefy covers have thermal pads underneath to help cool the SSDs.

The motherboard features dual LAN with 1 and 10 GbE connections, with the faster connection coming courtesy of an Aquantia AQC107. The -PG300 sports a debug LED and two 8-pin ATX power connectors that feed the 8+2 power phase underneath the beefy heatsinks. The board also features four PCIe 3.0 x16 slots and a single x1 slot, but you won't find any LED lighting on the motherboard. Instead, Supermicro provides two customizable RGB LED headers so you can roll your own solution. The company says that it is working on more elaborate lighting for future product generations. We also see the USB ports on the rear, including a USB Type-C port.

Supermicro also had the enthusiast-oriented C9X299-PGF on display, which features somewhat pared down specifications compared to the -PG300 but includes IPMI 2.0 support for out-of-band management. We don’t typically find that workstation/server-class feature on desktop PCs, so it makes for an interesting differentiator. The cooling solution for the VRMs isn't quite as robust with this model, but it only supports X-Series processors up to 165W TDP. You also get 5GbE connectivity with this model.

Supermicro's 300W rating for a Core X-Series CPU certainly raises many questions. We've heard of the 165W Core i9-7980XE pulling up to 550W with beefy watercooling and even closing in on 1000W with LN2, so it's no doubt the processors are power hungry. Intel's thermal paste chokes off most of the fun early, though. It's fun to imagine a 300W TDP processor coming our way, but it could be nothing more than marketing. Supermicro representatives wouldn't provide more information.

Supermicro's C9X299-PGF is already on the market. Its beefier C9X299-PG300 counterpart will join it later in Q1.

LAS VEGAS, NV -- The M.2 specifications have been great for consumer use, but lack some features large enterprise customers want in their pizza box server. The enterprise equivalent to M.2 is the U.2 specification, but that's built on old 2.5" disk drive form factor technology. Flash storage has moved beyond traditional form factors with success, but the consumer M.2 specification doesn't carry over well to the enterprise.

The two M.3 SSDs we've seen thus far from Samsung and now Adata feature a wider (30.5mm vs 22mm) width that gives manufacturers more surface area to mount components. The extra surface area gives the companies more options with NAND packages, host power failure capacitors, and larger controllers.

Adata's IM3P33E1 features a PCIe 3.0 x4 host connector that support hot plug. The drive will enter the market in sizes that range from 240GB to 1.92TB. This is lower than Samsung's, which reaches up to 16TB, but much more economical. Adata claims up to 3,200 MB/s sequential and 1,800 MB/s sequential write speeds over the NVMe 1.3 protocol.

The connector in the server doesn't use an angle when inserting the drive. This direct, straight through connector gives users hot plug capabilities without the use of a complicated mechanism. The only server we've seen with the M.3 form factor puts 34 drive sleds on the front of the 1U chassis. The server consolidates the drives vertically and fits more high-speed storage than possible with 2.5" U.2 ever could in 1.75" height server.

The pin out is the same between M.2 and M.3, but don't expect to use these in your desktop anytime soon. The two M.3 drives we've seen from Samsung and Adata use 12v instead of 5v. Even the name M.3 is not a technically correct term at this time—it's a specification that does not exist. Samsung first used the M.3 name at Flash Memory Summit 2016 but later shifted over to Next Generation Small Form Factor (NGSFF) after catching some heat from the committee that oversees the M.2 specification. With more than one company showing NGSFF drives, we should see some movement on the specification soon.

Introduction

Intel's Xeon E5-2600 v4 finds the company at the intersection of a waning desktop market and the exploding data center segment. Less momentum on the desktop side is naturally worrying; but being a leader in the high-growth enterprise space is a great place to be; Intel just happens to sell 99 percent of the world's processors destined for the data center. There is no single contender that poses an immediate threat to the company's dominance, though IBM's Power architecture and a budding Qualcomm initiative in China may gain ground over the long term.

The latest evolution in Intel's Xeon line-up is made possible by the Broadwell-EP architecture, manufactured on a 14nm process. That's a shrink of Haswell-EP's 22nm lithography. Many see Broadwell-EP as Intel's last attempt at satisfying the famed tick/tock cadence.

No doubt, Moore's Law is slowing to a crawl. The first precursors became evident in July of last year when the company indicated it'd push back its 10nm process to 2017. A recent 10-K filing merely formalized the obvious and inevitable.

Intel's filing indicates that it is migrating from the familiar tick/tock tempo to a slower process, architecture and optimization cadence. It appears that the challenges of shrinking transistors are becoming too difficult, and too expensive, to circumvent at such a rapid pace.

The new rhythm will lengthen the amount of time the company utilizes its 14nm and 10nm processes, which makes good business sense. Financially, it is better for Intel to recoup more of its investment into increasingly expensive tooling while extending its profitability window for each processor generation. There is no clear threat to the company's dominance, so its technological leadership position isn't at risk.

The Xeon E5-2600 v4 family wades into this new reality with a number of improvements that extend beyond more cores and cache (though it includes those as well). Perhaps that'll gives us some insight into Intel's plans moving forward.

Intel Xeon E5-2600 v4 Series

The Broadwell-EP microarchitecture increases the maximum number of cores/threads from 18/36 to 22/44, and also makes room for up to 55MB of shared L3 cache (up from 45MB). Intel still enables four channels of DDR4 memory, but increases the peak data rate to 2400 MT/s (a 15 percent improvement). Intel also added new memory features, such as support for 3DS LRDIMMs and DDR4 Write CRC (an enhanced form of error control). 

We're told the CPUs benefit from an IPC increase of 5.5 percent or so through a series of optimizations we'll cover on the following page, and are socket-compatible replacements for Xeon E5-2600 v3-based Grantley systems (LGA2011-3). Existing motherboards/servers will require a BIOS update, though. The existing C610 series chipset soldiers on, meaning that most platform features remain unchanged. You still get 40 lanes of PCIe 3.0 and two QPI 1.1 ports, for example.

The E5-2600 v4s offer a different base frequency and Turbo Boost setting for AVX and non-AVX functions, but Intel notably allows each core to operate in either mode without affecting the clock rate of other cores. In the past, every core ran at a lower base and peak Turbo Boost frequency if the cores were running a mix of AVX and non-AVX code. That restriction is no longer in place.

MORE: Best CPUs
MORE: Intel & AMD Processor Hierarchy
MORE: All CPU Content

Our Xeon E5-2697 v4 sample features 18 Hyper-Threaded cores and provides a non-AVX base frequency of 2.3GHz, peaking at 3.6GHz under Turbo  Boost. Running AVX code, the base drops to 2GHz, while Turbo can still take you to 3.6GHz.

The Product Stack

The fourth-generation E5-2600 series features 22 models designed for a wide range of workloads. Intel groups the processors into High Core Count (HCC), Medium Core Count (MCC) and Low Core Count (LCC) segments ranging from four cores and 10MB of LLC to 22 cores with 55MB of cache.

New Intel 3D NAND SSDs

All of the processing power in the world is worthless if your CPU is waiting on the storage subsystem. Intel helped foment the SSD revolution when it introduced its first datacenter-oriented drives. Of course, the move was a strategic one. Intel makes more money selling storage. In turn, SSDs help unlock the potential of multi-core processors, encouraging more Xeon business.

Intel and Micron produce NAND in the jointly operated venture called IMFT, which recently announced 3D NAND-based products. Intel chose to launch two drives based on the technology, its DC P3520 and DC P3320 SSDs, in tandem with Broadwell-EP.

The DC P3320 series is news to us, but the DC P3520 is not. We discovered a document back in August that foretold the release of Intel's DC P3608, and it also included pertinent information on the DC P3520. Intel briefed us on the DC P3320 during its Broadwell disclosure. However, it kept information about the DC P3520 to itself. Don't worry, though. We have the scoop on all of Intel's new SSDs, including dual-port NVMe-based offerings, on page three.

First, let's take a closer look at the Broadwell-EP microarchitecture.

Broadwell-EP Architecture

The Broadwell-EP line-up is based on three different die configurations with modular designs. The HCC die measures 18.1x25.2mm and comprises ~7.2 billion transistors. The architecture itself still employs two full rings per HCC die, but now it's symmetrical. In Haswell-EP, the ring on the right serviced two additional cores, creating asymmetry.

Here, Intel connects both bidirectional rings to 12 cores each, and it disables an equal number of cores per ring to create SKUs with fewer cores. As an example, the flagship 22-core Xeon E5-2699 v4 has 11 active cores per ring. As you work your way down the stack, two cores at a time are turned off, one from each side, along with their corresponding slices of last-level cache. That's how Intel creates models with less L3, too.

Each active core is associated with 2.5MB of LLC cache that is shared across its ring, and any core can address any part of the cache. The advantage of two distinct rings is more efficient scheduling; everything that happens on one ring is independent and occurs without any interference from the other ring. Routing ring traffic intelligently, and in the correct direction, is naturally quite important; a transaction on the ring can take up to 12 cycles (depending on how far it has to travel). There's intelligence built in to address this. Without it, if a core needed information in cache to the "south" of it and the traffic went north, that request would have to make a complete loop. Instead, the scheduler correctly routes traffic south, yielding faster access to data in the cache.

Balancing a workload between two rings also reduces the number of cycles that would be required to navigate one larger ring. The only caveat is that routing traffic between rings requires a trip across the buffered switches connecting them at the top and bottom, which incurs a (roughly) five-cycle delay. Each ring has access to its own memory controller (bottom), but only the ring on the left has access to the QPI links and PCIe lanes (top).

The MCC die measures 16.2x18.9mm and has ~4.7 billion transistors, while the LLC die measures 16.2x15.2mm and employs ~3.2 billion transistors.

Intel drops the number of cores per ring from 12 to 10 on the MCC and LCC configurations, but continues to employ a bidirectional ring structure. The MCC's partially severed ring even gets an additional memory controller. Then Intel does remove the second ring's last traces for the Low Core Count (LCC) die, eliminating it and the other memory controller. This also gets rid of any reason to have the buffered switches, which connected the two rings on the larger dies.

LCC-based models can still address four DDR4 memory channels through the single controller, illustrated by the four arrows emanating from that piece of logic. This results in a small loss of throughput, since there isn't a second memory scheduler to help service transactions. But Intel doesn't quantify the extent of the performance impact.

Performance Boosting Technologies

Broadwell-based CPUs boast a roughly 5.5% IPC boost compared to Haswell. The most notable improvements affect floating-point instruction performance, and include a reduction in Vector FP multiply latency from five cycles to three, improvements to the Radix-1024 divider, split scalar divider and hardware assist for vector gather operations (60 percent fewer).

Other compelling additions include virtualization-centric features like posted interrupts, which reduce VM enter/exit latency by batching the interrupts, and page modification logging, minimizing the overhead of VM-based fault tolerance through rapid checkpointing.

Intel also employs Transactional Synchronization Extensions (TSX) to boost performance, and its new Hardware Controlled Power Management purportedly cuts power consumption. We'll put that claim to the test on page eight. 

Orchestration And Security Features

Intel's Resource Director Technology provides enhanced telemetry data, which allows administrators to automate provisioning and increase resource utilization. This includes Cache Allocation Technology, Code and Data Prioritization (CDP), Memory Bandwidth Motioning (MBM) and enhanced Cache Monitoring Technology (CMT).

You also get a spate of enhanced security features, including faster data encryption and decryption, network security and trusted compute pools through Crypto Speedup (ADOX/ADCX), a new random seed generator (RDSEED), Supervisor Mode Access Prevention (SMNAP) and Virtualization Exception (#VE) technology.

Models And Pricing

Intel Test Platforms And How We Test

How We Test

We tested the Broadwell-EP-based Xeon E5-2697 v4 on both an Intel Software Development Platform server and the production-ready Supermicro SYS-1028U-TN10RT+. We tested the Haswell-EP-based Xeon E5-2699 v3 and E5-2643 v3 on the Intel Software Development Platform. And we tested the Ivy Bridge-based (v2) CPUs in Intel's Server System R2208GZ4GC. 

We benchmark with the open source Linux-Bench script, which is available on Linux-Bench.com and GitHub. ServeTheHome and others in the open source community maintain it. The suite runs from an Ubuntu 14.04 LiveCD either on local storage or through a KVM-over-IP connection. The script installs dependencies and runs several well-known independent open source benchmarks that characterize CPU performance.

Intel "Wildcat Pass" S2G3SY1Q Server

Intel sent us a pre-production Grantley-R EP S2G3SY1Q (Wildcat Pass) Broadwell Qualification system for our tests. The 2U test bed came with two Xeon E5-2697 v4 CPUs with 18 Hyper-Threaded cores and 45MB of shared cache apiece (that's 2.5MB of last-level cache per core). These beefy CPUs feature a non-AVX base clock of 2.3GHz and a maximum Turbo Boost frequency of 3.6GHz. The E5-2697 v4 offers a base clock of 2.0GHz and a Turbo Boost ceiling of 2.6GHz under AVX-optimized workloads.

The test platform features Intel's C610 chipset family and includes eight 32GB SK hynix DDR4-2400 DIMMs (HMA84GL7AMR4N-UH). Two riser cards enable PCIe connectivity, but weren't used in our testing.

Intel provides this server for use as a software development platform; it's not designed for use in a production environment. As such, it lacks some of the features that facilitate redundancy, such as dual PSUs. One of the PSU bays is covered, but the other houses a single 900W power supply.

Intel Server System R2208GZ4GC

An Intel R2208GZ4GC functions as one of the workhorses in our enterprise storage lab, and it has the scratches, bumps and bruises to prove it. At its heart, the server features the S2600GZ motherboard (C602 chipset) housed in a production-class chassis with the requisite redundant and hot-swappable fans, along with dual hot-swappable 750W power supplies.

Two riser cards have seen more than their fair share of RAID controllers, HBAs and PCIe SSDs, but again, they weren't required for today's benchmarks. We installed 64GB of Kingston DDR3-1600 memory in 8GB modules. Other notable platform features include quad 1GbE connections and an RMM4 module for remote management.

Supermicro And NVMe RAID Testing: 3 Million IOPS And 21 GB/s

Supermicro SuperServer SYS-1028U-TN10RT+

We already had the slender 1U Supermicro SuperServer SYS-1028U-TN10RT+ in our enterprise storage lab, so after a quick BIOS update we slapped in the Xeon E5-2679 v4 CPUs and went to work.

Don't let its diminutive stature fool you; the SuperServer is a little powerhouse. It features 10 hot-swap NVMe bays that we populated with ten 400GB 2.5" Intel DC P3700 SSDs. Normally, 16 counter-rotating fans provide copious airflow over 24 Samsung 16GB DDR4-2133 ECC DIMMs, but we removed 16 of them to level the playing field. Even with 384GB of DDR4 at our disposal, we are not populating the server to capacity; it supports an amazing 1.5TB of DDR4 ECC LRDIMMs. 

The NVMe SSDs mate to the PCIe backplane, which Supermicro routes to a pair of switchless expansion cards in the rear. These cards provide direct PCIe 3.0 x4 connections to the CPU, enabling incredible performance. The back of the chassis features two 10GbE ports and one management port, along with dual Titanium-rated 1000W redundant power supplies.

Need a slice of parallel processing in the storage sandwich? You also get three auxiliary eight-pin power connectors, allowing the server to accommodate either two full-height full-length graphics cards, or one double-width board and a low-profile offering.

We configured the 10 Intel NVMe SSDs into a RAID 0 volume (mdadm) in CentOS 7, demonstrating how the E5 2697 v4's 36 threads come into play under heavy I/O operations. The colored spaghetti strings in the picture above indicate the load of each core during a 32-thread 4KB random read workload.

Incredibly, we're pushing just over 3 million IOPS during this capture with an aggregate CPU utilization of 90.9 percent. To put that into perspective, you're seeing nearly 12 GB/s of random throughput. Now you see why very fast multi-core CPUs are so important to extracting maximum performance from NVMe-based SSDs in servers.

These results are hard to beat; we recently took 64 of the fastest PCIe SSDs out for a spin in 64 PCIe SSDs, 120TB Of Flash And One Stop Systems: The FSA 200 Review. The FAS 200 topped out with 3,055,735 IOPS in a system that monopolizes 5U of rack space. Meanwhile, the 1U Supermicro server musters even more random read performance thanks to its switchless design. We reach 814,351 IOPS during the 4KB random write workload, which is excellent scaling considering that Intel specs each drive at 75,000 random write IOPS.

The SuperServer continues to impress with a beastly 21 GB/s of sequential read throughput (yes, gigabytes), easily surpassing the 13 GB/s we achieved with the FAS 200. 

It is hard not to like this much power packed into such a small footprint; it's the personification of both performance and storage density. Now that we have seen what Intel's current-gen SSDs can do, let's take a look at the next generation.

Intel's 3D NAND SSD Debut: DC P3520/P3320 And DC3700/3600

DC P3520 And DC P3320 Series SSDs

Intel is simultaneously launching four datacenter SSDs, which are particularly interesting because the DC P3520 and DC P3320 both employ the company's long-awaited 3D NAND. The jointly developed IMFT flash sets a new standard for density with 32 layers and 256Gb per MLC die and an impressive 384Gb per die of 3D TLC.

The new NAND enables incredible density. We'll see up to 512GB for a 16-die MLC package and up to 768GB in a 16-die TLC package. IMFT 3D NAND employs floating-gate technology, which is quite a bit different than what other NAND fabs are doing with their Charge Trap Flash (CTF). Intel has a long history with floating-gate transistors, and the company believes it can extract more performance and endurance compared to competing CTF designs. IMFT NAND also puts most of the CMOS under the array, which boosts efficiency and density.

We were able to snap a few pictures of the DC P3520 at Intel's Broadwell-EP press briefing. As you can see, the SSD is physically almost identical to the company's existing DC P3700 SSDs.

The DC P3320 comes in capacities as large as 2TB, matching its counterparts based on planar NAND. This is surprising considering the higher density possible with 3D NAND. But Intel indicated that the new design features the same controller as the previous-gen planar NAND-based products. That controller can only address up to 2TB due to DRAM management limitations.

Intel sells the DC P3320 in both 2.5" and add-in card form factors with 450GB (2.5"-only), 1.2TB and 2TB of flash. The company rates its DC P3320 for up to 365,000 random read and 22,000 random write IOPS. Those write numbers are similar to the low-endurance DC P3500 (23,000 IOPS), but the random reads are not nearly as fast as the DC P3500's 430,000 IOPS. The DC P3320 offers up to 1600 MB/s sequential reads and 1400 MB/s of sequential writes, which also lags its NVMe-based predecessors.

Intel attributes the lower performance to a loss of parallelism due to higher-capacity 3D NAND dies. Future models will sport an improved controller that manages more DRAM, thereby accommodating more capacity and parallelism, which ultimately improves performance. Intel doesn't get specific, but indicates that the DC P3320 will feature a lower price than the DC P3500 and a similar write endurance rating (0.3 DWPD for five years).

Intel continues its focus on performance consistency, touting the end-to-end data path protection that provides a resilience to silent data corruption. Intel is one of the few storage vendors confident enough to share field reliability data, and it continues to maintain an Annual Failure Rate (AFR) below 0.44%.

Intel also mentioned the new DC P3520 during its briefing. However, the company did not share any performance details. We did uncover a document last August that accurately foretold the DC P3608's release though, and it included information on the DC P3520. The slide above is a bit dated at this point; its listed availability was originally anticipated in Q4 2015. But other information in the same document proved true, so we are fairly confident that it still reflects the projected specifications. At the very least, Intel hinted that the DC P3520 will be a performance-oriented model, which agrees with the numbers listed above.

Intel DC D3700 And DC D3600 Series SSDs

Intel also announced dual-port active-active NVMe-based SSDs at the event, which closely resemble the 2.5" DC P3700 NVMe SSD we recently tested and the 2.5" DC P3320 pictured above.

SAS-attached SSDs still enjoy the advantage of two ports, which provide an active-active connection for High Availability (HA) features like multi-path and fail-over. These are extremely important in mission-critical applications, and NVMe's original lack of dual-port functionality hampered its adoption in many of the long-held SAS-based SSD bastions, such as all-flash arrays. In fact, we dinged the 2.5" Intel DC P3700 NVMe SSD for that exact reason in our review earlier this month.

The NVM Express consortium added dual-port support to later revisions of the standard, and Intel's DC D3700 and DC D3600 SSDs provide two PCIe 3.0 x2 connections into compatible backplanes. Although this allows multiple hosts to access the SSDs, it also cuts performance in half, from a top speed of 470,000 random 4KB read and 95,000 write IOPS, and 2100 MB/s sequential read 1500 MB/s sequential write throughput.

Intel's new NVMe SSDs are only available in a 2.5" form factor, and they require special backplanes to take advantage of the dual-port functionality. You'll only find the DC D3700 at 800GB and 1.6TB capacity points, though you get 10 DWPD of endurance. The DC D3600 is available in 1 and 2TB capacities with a 3 DWPD rating.

Features supported by the drives are similar to what you get from SAS-based SSDs, including reservations, scatter/gather lists, multiple namespaces and in-controller memory buffers. Intel indicates that the mulitple namespace capability runs in hardware, which is better than SAS' software-based approach, increasing efficiency.

Benchmarks

Unix Bench 5.1.3

Unix Bench is an open source GPLv2-licensed test suite that originated in 1983 at Mondash University. Byte Magazine then updated and expanded the test suite. It includes nine tests that quantify various aspects of system performance, but we focus on the Dhrystone and Whetstone metrics. The former focuses on string handling; there are no floating-point operations. Whetstone measures the speed and efficiency of floating-point operations typically employed in scientific applications.

Both Xeon E5-2697 v4-based systems take a tremendous lead over the competition in our multi-threaded Whetstone test, which is impressive because the -2699 v3 features an identical core count, cache and clock rate. Apparently a big boost in IPC throughput, among other optimizations, delivers a strong performance improvement.

Conversely, the E5-2697 v4 comes crashing down to the bottom of the single-threaded Whetstone test. Both v4-powered servers muster more performance in the Dhrystone test, but still fall behind the Xeon E5-2643 v3. This is not entirely surprising; the eight-core -2643 v3 sports a higher 3.2GHz base frequency and a slightly better 3.7GHz max Turbo Core clock rate.

The Xeon E5-2643 v3's lower score in the single-threaded Whetstone test is unexpected. These results come from an average of three runs, but we retested the -2643 v3 extensively to ensure its outcome was repeatable.

NAMD Scalable Molecular Dynamics

NAMD is a parallel molecular dynamics code that performs high-performance simulations of large biomolecular systems. It's designed to test the boundaries of parallel computing by scaling up to 500,000 cores. The Theoretical and Computational Biophysics Group at the University of Illinois at Urbana-Champaign developed and maintains the project, which is the recipient of several prestigious awards.

The Xeon E5-2697 v4 easily finishes with a comfortable lead over competing CPUs in this parallel workload. The previous-gen -2643 v3 lands in last place, which aligns with our expectations due to its lower core count in comparison to the other contenders.

NAS Parallel Benchmarks

The NAS Parallel Benchmarks (NPB) are a set of programs developed by the NASA Advanced Supercomputing Division to evaluate the performance of parallel supercomputers. The test suite characterizes Computational Fluid Dynamics (CFD) applications, and NASA designed it to measure performance from smaller CFD applications up to "embarrassingly parallel" operations. 

The two Xeon E5-2697 v4 configurations are closely matched in the 64x64x64 BT test, but the -2699 v3 noses in between them with an impressive score as well.

The Broadwell-EP-based processors in Intel's development platform perform as we'd expect them to, while the same Xeons in Supermicro's enclosure lose quite a bit of steam. This suggests that the test's more complex portion is influenced to some extent by system memory performance. After all, the Intel chassis is rocking 2400 MT/s DDR4, while the Supermicro platform uses more pedestrian 2100 MT/s RAM. 

The M/ops per thread results indicate the performance of single cores during the test; they align with our expectations.

Redis

REmote DIctionary Server (Redis) is a popular in-memory data structure store that is used as a database, cache and message broker. It's dependent on both CPU performance and memory bandwidth. Redis is open source (BSD-licensed) software.  

The Xeon E5-2697 v4s in Intel's chassis take the lead thanks to higher memory bandwidth than the Supermicro-based platform. We also tested the -2643 v3 and -2699 v3 processors in the Intel server, which explains their stellar performance in this benchmark. Clock rate clearly comes into play as well (the -2643 musters impressive performance). Meanwhile, the second-gen E5s based on Ivy Bridge languish at the bottom of the chart due, in part, to their DDR3 memory subsystem.

Open SSL

SSL (Secure Sockets Layer) is the go-to technology for securing encrypted links between a server and client. It's an important protocol for every level of the Web in today's security-conscious climate.

According to Intel, encryption/decryption performance was one of the Broadwell team's targets during the design phase, and the Xeon E5-2697 v4 demonstrates the results of those efforts. As you can see, our OpenSSL test utilizes multiple cores well.

Additional Benchmarks

Sysbench CPU

Sysbench is a widely used suite that characterizes CPU, memory, file I/O, mutex performance and MySQL performance. We focus on the CPU test, which measures the amount of time required to verify prime numbers in both single- and multi-threaded workloads.

Both Xeon E5-2697 v4-powered servers provide impressive performance, beating most of the competition in the threaded component of this test. But Intel's -2699 v3 spoils the party by reporting a slightly faster result. The v4s fire back, overtaking the Xeon E5-2699 v3 in single-threaded performance, even besting the E5-2643 v3 as well.

Stream

Stream is a relatively simple test developed by Dr. John D. McCalpin. It measures the sustainable memory throughput of a given system in MB/s.

The Intel validation server employs speedy SK hynix DDR4-2400. As such, it leads our bandwidth results. The Xeon E5-2699 v3 employs dual memory controllers, which apparently are not as efficient as what Broadwell-EP brings to bear. They trail, even though they are tested on the same Intel motherboard.

We also tested the Xeon E5-2643 v3 on this board, and it trails significantly. The eight-core die only employs one memory controller, which just can't keep up.

C-Ray 1.1

C-Ray is a raytracing benchmark designed to reside entirely inside of the CPU caches, thus eliminating RAM and disk I/O overhead during the measurement window. The test focuses purely on floating-point performance during rendering and runs on multiple threads.

The -2697 v4s and -2699 v3 are locked in a three-way tie for the lead, while the second-generation E5s fall to the rear. Because this test scales well across multiple threads, the eight-core Xeon E5-2643 v3 is quickly outclassed, despite high clock rates.

7-Zip

7-Zip is open source software that measures compression and decompression performance, which can be a key capability for storage and networking applications.

The only surprise during this test is just how well the 18-core v4s fare compared to the Xeon E5-2699 v3. Meanwhile, all of the second-gen E5s we tested fare dismally, illustrating the benefits of a more modern platform. Most enterprises will be upgrading from Sandy/Ivy Bridge to Broadwell, and this benchmark does a good job of highlighting the type of performance boost to expect.

HardInfo

HardInfo provides granular system information, and includes a suite of six benchmarks that measure CPU performance. It's easily accessible and comes as a standard component in many Ubuntu desktop systems. We include these tests because they allow our Linux brethren to easily run comparative benchmarks.

The v4s lead in FPU raytracing, but fall behind their predecessors in the Fibonacci test. This generational face-off gets a little more heated during the FPU FTT tests, and the Xeon E5-2687 v4 in Intel's development platform dominates the Cryptohash test.

Power Consumption

There is no way to understate the impact of power consumption and efficiency on the modern datacenter. Power use is the killer of all things bottom-line. Datacenters worldwide consumed an estimated 416.2 terawatts (416.2 trillion watts) of electricity in 2014, which is more than 182 entire countries (out of 192 globally). Even more alarming, datacenter power use is quadrupling every two years, and a recent Japanese study concluded that at today's rate, its datacenters will consume all of the country's electricity output by 2030.

The pressure to reduce power consumption is an overbearing burden, but at the same time, the demands for more processing power are increasing every year. Deploying more efficient CPUs simultaneously increases performance per watt, frees up floor space and relaxes cooling requirements. All three of those variables function as multipliers that can either reduce or inflate datacenter expenditures dramatically.

Intel took several important steps to reduce power consumption with its previous-generation Haswell-EP products, including the addition of an on-package power delivery system. It also added per-core P-state control and moved to DDR4 memory, which helped reduce platform power use, further improving efficiency.

The Xeon E5-2600 v4 family offers these same advantages, but also brings about the introduction of Hardware Controlled Power Management (HWPM). Under normal circumstances, most servers accept hints from the operating system that indicate when it is appropriate to adjust the power state. Unfortunately, this process is slow in relation to how fast a CPU can make decisions internally, and not all software developers are utilizing the feature.

HWPM turns power management over to the CPU, offering up to four power profiles that optimize the server for each use-case, which you can specify in the BIOS. The CPU then adjust power state settings dynamically, depending on the chosen profile. Not only is this process faster, but it's also compatible with all operating systems (the CPU simply ignores the OS hints). Intel is in the early stages of HWPM development, but expects the feature to evolve rapidly.

Linux-Bench Power Consumption

We measured power consumption during the Linux-Bench script, which provides a nice comparison point for each system as the test progresses. The first slide provides a view of the test servers in one image. Then, we provide two additional slides that paint a clearer picture.

The Supermicro platform is unsurprisingly more efficient than Intel's software development platform. After all, it's a production-class system with 80 PLUS Titanium-rated PSUs.

The Xeon E5-2699 v3 CPUs consume more power than our Broadwell-EP samples in the same Intel development platform, which speaks volumes given their similar core count. The -2643 v3 consumes less power by virtue of its lower core count. This is an important consideration; it is not wise to provision excess CPU resources that exceed the requirements of the workload. It's better to pick the right CPU for your application. 

Power Load And Idle

We measured peak power consumption during a Linpack run to characterize each platform's use. Again, Supermicro's configuration is more efficient than Intel's in both maximum and idle consumption. We also recorded the watt-hours consumed during the entire Linux-Bench script, and found the Supermicro platform offering a more refined efficiency story. The second-gen Xeon E5s do use less power, but they're also a lot slower.

Conclusion

Intel's switch to the process, architecture and optimization cadence is a clear indicator that the heady days of Moore's Law is drawing to a close. Over the last few decades, Intel picked the low-hanging fruit from the transistor tree as it moved forward at an incredible pace, but now we're finding each processor generation offering smaller and smaller performance improvements.

The majority of datacenter and enterprise customers are locked into three- or five-year refresh cycles due to maintenance contracts, and as those contracts expire, they upgrade to the newest platforms. This means that the majority of Broadwell-EP customers will be migrating from either Sandy Bridge or Ivy Bridge, and not replacing Haswell-EP-based processors with newer Broadwell-EP models.

As a result, they'll reap the benefits of not only Intel's Broadwell-EP design, but also the mature C610-series chipsets, which continue to find use with the fourth-generation Xeon E5s (updated with fresh firmware, of course). That platform controller hub family merely serves to support a well-endowed host processor. The PCH has limited PCIe 2.0, some built-in USB 3.0 and quite a bit of SATA connectivity. Really, it's the Xeon's big PCIe 3.0 controller, fast QPI links and DDR4 memory controllers (now able to accommodate 2400 MT/s modules) that do the heavy lifting.

The CPU's key advances include a top-end model with four more physical cores, a roughly 5.5 percent improvement in IPC throughput and a last-level cache increase from 45 to 55MB. Our benchmarks show the Xeon E5-2600 v4 serving up nice performance increases through most of the suite. Numerous platform-level improvements offer massive performance advantages to anyone migrating from old Sandy Bridge and Ivy Bridge servers. On the flip side, there's probably not enough reason for most businesses to ditch Haswell-EP in favor of Broadwell-EP, unless they really need a specific new feature.

Speaking of new features, they appear to be an increasing focus at Intel as the glory days of massive performance gains fade into history. In lieu of big speed-ups, Broadwell-EP includes some extras designed to enhance performance and usability in a variety of applications. VM-centric optimizations should enjoy considerable use in the wild, and the new orchestration tools allow administrators to monitor, manage and optimize with a heretofore-unseen level of granularity. Increased performance and manageability in security-related tasks through faster encryption/decryption, a new random seed generator, SMNAP and Crypto Speedup will come in handy as well.

Intel is wisely using its Broadwell-EP launch as a springboard to introduce a number of high-powered SSDs, too. The meteoric rise of solid-state storage in the datacenter is making it easier to fully utilize potent multi-core CPUs, and matching speedy NVMe-based drives up to the latest in processor technology proves key in bleeding-edge applications, as evidenced by our NVMe RAID tests.

Many enthusiasts are looking to Skylake as the next truly revolutionary advance in processor technology, and there are predictions that it will support 3D XPoint additives and on-die Omni-Path adapters. Sure, Skylake is exciting. But the 14nm node might have more to offer in the near term as well.

Intel recently displayed Arria 10 FPGAs and what appeared to be a Broadwell die on the same package at the Open Compute Summit. It announced that products with this design will ship in 2016. The company's recent Altera purchase looks like it's bearing fruit already, and we may see an interesting marriage of CPU and FPGA on the same die during the transition to process, architecture and optimization.

In the meantime, Intel's Xeon E5-2600 v4 family provides an attractive upgrade path that offers gains in performance, cores, cache and power consumption that will satisfy the vast majority of users, and in particular, those migrating from Ivy and Sandy Bridge. 

MORE: Best CPUs
MORE: Intel & AMD Processor Hierarchy
MORE: All CPU Content

Paul Alcorn is a Contributing Editor for Tom's IT Pro and Tom's Hardware, covering Storage. Follow him on Twitter and on Google+. Follow us on Facebook, Google+, RSS, Twitter and YouTube.

Introduction

Motherboard comparisons are usually a race to the finish, with each firm trying to beat its rivals in performance, overclocking or performance and overclocking per dollar. Those are all great metrics when the hardware is there, but we were shocked when some of it wasn’t. Even though all of the big names in motherboards seemed to have production-ready samples two months ago at Computex, the reality is that most of them were far from ready as little as a week ago.

Part of that, we’re told, was due to Intel’s continuous string of minor core firmware upgrades. That’s the core onto which most companies add their customizations, including the GUIs that allow us to overclock easily (or with great difficulty, depending on that firm’s design). In the end, we received a bunch of motherboard samples over the past two weeks. But the firmware for most of them wasn’t ready until days ago. That’s a little late for trying to time a platform round-up to coincide with the CPU review.

And then there was the race for memory. Overclocking is a huge part of an enthusiast motherboard review, and better memory allows motherboard to get closer to their maximum DRAM data rate. In spite of constant improvements in memory technology, a single set chosen for the first review in a series tends to stick around until we’ve finally reached the last comparison board. This lets us compare competing platforms without going back and retesting old samples.

Dual-channel kits are easier to get “up to speed” than quad-channel, so I wasn’t surprised when various manufacturers started announcing super-high data-rates for Skylake. G.Skill was first to offer a 2x 8GB DDR4-3400 kit, followed by a 4x 4GB DDR4-3466 configuration when I countered with comments about memory multipliers, controller ratios and two-vs.-four-DIMM overclocks.

As I waited, Corsair introduced me to its own DDR4-3400 memory. As soon as I’d set up the backup plan, G.Skill let me know that it had shipped a 4x 4GB DDR4-3600 kit. Corsair said it would send a kit anyway, and its DDR4-3200 kit arrived before G.Skill’s 3600 MT/s parts. Caught between two manufacturers, I decided to use the higher-rated G.Skill parts for motherboard tests and give the tighter-timed Corsair kit its own, separate review.

Now that the hardware is up to the starting line, I could go into great detail about the platform itself. Fortunately, our CPU testers have already taken care of that. The biggest thing I’d like to mention is that even though we now have a bunch of PCIe 3.0 lanes available from the PCH, we’ve only doubled DMI bandwidth compared to the Z97. According to internal documents, a mere four lanes of full PCIe 3.0 traffic can consume the entire link between the Z170 chipset and host processor, so users shouldn’t think that there isn’t a penalty for going crazy with PCIe-based SSDs, SATA-E, RAID cards, extra graphics cards and the like. This isn’t a storage server solution, and if we think of the Z170 as little more than a four-lane to 20-lane PCIe hub with a few extra built-in controllers, we can probably keep our bandwidth use in check.

Keeping those limits in mind, here’s what the first three motherboards have to offer:

Technical Specifications

Prior to sending its retail-boxed Z170 Extreme6, ASRock sent a pre-production Z170 Extreme7+, along with processors, as an overclock training board. As was the case with the memory, another vendor offered first and delivered last. I could have been tied up for days figuring out which CPU to use but, fortunately, my two Core i7-6700K samples proved nearly identical in core overclocking, DRAM overclocking, power and heat.

MORE: Best MotherboardsMORE: 
How To Choose A Motherboard: A Guide For BeginnersMORE: All Motherboard Articles

MORE: Latest Motherboard NewsMORE: Motherboards in the Forums

ASRock Z170 Extreme6

Shrouded in black atop a black PCB, the Z170 Extreme6 minimizes its use of gold accents (which reflect other colors from the background). Like most enthusiast-class motherboards, the Extreme6 features an inconspicuous CLR_CMOS button so that overclockers can return their system to a bootable state without opening the case. In deference to a few of our more “enthusiastic” readers, the I/O panel features six USB 3.0 and two USB 3.1 (Type A and Type C) connections, and no USB 2.0 or VGA. It keeps the PS/2 port in defiance of those enthusiasts, since ASRock knows certain power users prefer their ancient keyboards and mice. And, if you really need a VGA output for diagnostic purposes, you can adapt it from the DisplayPort connector.

If all you ever wanted to do with your enthusiast-class board was push a bunch of displays, the Z170 Extreme6 complies with three x16-length and three open-ended x1 slots. The open ends allow longer cards to be placed in interfaces wired to fewer lanes.

The top x16 slot gets all of the CPU's available lanes, and four two-lane switches allow eight of those lanes to be split off to the second “x16” slot for x8/x8 mode. The third x16-legth slot gets its four lanes from the Z170 PCH, which prevents its use in an SLI configuration (though AMD still allows its use in CrossFire).

One reason we don’t see any oddball PCIe lane limitations is that the Z170 Extreme6 has but one M.2 slot. That slot still steals a PCIe pathway from one of the SATA-E connections or, if you’re using a SATA-based M.2 card, a SATA port. ASRock tries to make up for any of those potential disappointments by adding an ASM1061 controller to host two more SATA 6Gb/s drives.

The Z170 Extreme6 is the least-expensive board in today’s line-up, though that doesn’t make it cheap. ASRock still outfits it with an O/C-friendly 12-phase CPU voltage regulator, a digital boot code display, power and reset buttons to ease testing outside of a case and, as with most ASRock boards, dual socket-mounted firmware ROMs with a physical jumper to select the active IC. We also find six four-pin fan headers with five independent control maps.

The Z170 Extreme6’s single USB 3.0 front-panel header is located above the top card slot to avoid interference with just about anything, and its SATA/SATA-E ports all face straight forward to tuck under the leading edge of add-in cards. That means the only conflict—front-panel cable headers and a graphics card in the third x16-length slot—can usually be “resolved” by smashing the cables flat. The closest thing remaining to a layout problem is that the front-panel audio cable is a little close to the bottom-rear corner. But that’s primarily a case problem, since boards have been designed this way since the early days of ATX.

Even though the Z170 Extreme6 supports eight SATA drives, it has only four SATA cables. ASRock also includes a two-drive SATA power cable, which attaches to the motherboard, to allow drive power to be disabled via its HDD Saver software. The motherboard includes an SLI bridge and a USB 3.1 bay adapter, too.

That last value-add certainly got our attention. The Z170 Extreme6 comes bundled with a front-panel USB 3.1 adapter that serves anyone elite enough to have an empty 5.25” bay with a Type A and a Type C port via an ASM1142 controller connected through an included SATA-E cable. The only part of this we find a little disappointing is that, by using a full SATA-E cable to access only its PCIe connection, users “waste” the two adjoined SATA ports.

Software And Firmware

ASRock A-Tuning includes several overclocking profiles, from 4.4GHz at 1.28V to 4.8GHz at 1.39V, which are repeated from firmware settings, configured through firmware settings and thus require a reboot to alter firmware settings. Two “OC Gear” settings of 4.2-4.6GHz at up to 1.28V and 4.2-4.7GHz at up to 1.375V are also configurable using the “Advanced Turbo” functions. Finally, the board’s tuning and stability test cycle returned a 4.3GHz O/C at 1.28V based on our processor’s capabilities.

Manual tuning is available through the OC Tweaker menu within the range supported by hardware, which is also available in firmware, and the results are easily found under A-Tuning’s “System Info” tab.

Clicking the “System Browser” tab within A-Tuning’s “System Info” menu brings up an image of the Z170 Extreme6, outlining installed devices in yellow. This is particularly handy when trying to figure out whether a device was detected but not properly configured, which occasionally happens when overclocking.

Four chassis and two CPU fans can each get their own custom fan profile under A-Tuning’s “Fan-Tastic Tuning” menu. Other menus include a tech service online app and an auto-start with Windows checkbox.

Z170 Extreme6 firmware opens to a basic GUI with items like a firmware-based System Browser (similar to the software version), Instant Flash for updating firmware from a USB drive, Internet Flash for downloading firmware from ASRock’s server sans-OS, EZ OC auto-overclocking, RAID mode enabling, XMP enabling and boot priority.

Custom overclocking requires a press of the F6 key to enter advanced mode, where the OC Tweaker menu provides submenus with manual clock, voltage and timing controls. It also has the “Optimized CPU OC” settings discussed in the software section, above.

The CPU section of OC Tweaker provides clock, ratio, wave form detection and power controls. We were able to get our Core i7-6700K to 4.6GHz at 1.30V.

The Z170 Extreme6 offers more memory tuning options than most of us could ever hope to master, all the way down to quaternary timings. Fortunately, a tuner can chose to control only a few of these and let the motherboard’s programing determine the rest. This motherboard sample pushed our DRAM all the way to DDR4-3500 at a mere 1.35V.

Voltage controls are on a separate menu, where we found Level-2 “Load-Line Calibration” needed to keep our 1.30V core setting stable under excessive eight-thread Prime95 loads. The memory voltage regulator sneaks in about 25mV more than ASRock reports, so that the 1.35V DRAM test voltage discussed above required the board’s 1.330V setting.

MSI Z170A Gaming M7

MSI is jumping onto Apple’s bandwagon in the naming of its USB 3.0 ports, calling them USB 3.1 Gen 1. Both names are considered valid, yet the newer one seems intended to confuse buyers into thinking they’re getting more than they paid for. By traditional names, the Z170A Gaming M7 includes a pair (Type A and Type C) of rear panel USB 3.1 ports (10 Gb/s), a pair of USB 3.0 ports that are confusingly given the same color, three USB 2.0 ports, two HDMI ports, a DisplayPort output, GbE, a CLR_CMOS button and the traditional five analog/one optical audio connections.

The analog audio jacks are gold-plated to assure oxide-free connections, the network port is lit from the inside and the CLR_CMOS button lights up when engaged. You’ve probably noticed that one of the USB 2.0 ports is vertical, and that’s because MSI wants it to stand out as supporting its USB Flashback function. That’s right. MSI adds a CPU-free, DRAM-free firmware upgrading feature just in case you try to install a newer-model CPU that requires a firmware upgrade to boot.

MSI outlines the two CPU-driven PCIe x16 slots with aluminum braces that, in turn, are supported by the plastic beneath. These may help prevent the slot from opening up in the middle (losing contact with the card). But they do little to prevent a heavy card from ripping the slot off the board during the rough handling that occasionally occurs when shipping a complete system. Flipping the board, I only found one extra attachment point per slot.

As with the other boards in today’s round-up, those two slots share 16 lanes in either a x16-x0 or x8-x8 configuration via automatic switches that detect the second card. The third x16-length slot is tied via four lanes to the Z170 PCH, supporting CrossFire but not three-way SLI.

All four PCIe x1 slots are open ended, but using them forces the four-lane slot to a single lane. This probably doesn’t matter to you if you’re using a bunch of single-slot graphics cards to drive displays in 2D mode though.

The top board by both price and features in today’s comparison, MSI's Z170A Gaming M7 has two PCIe x4 M.2 slots. Though both slots support PCIe 3.0 x4 connections, total bandwidth is limited by the chipset’s 32 Gb/s DMI interface. SATA mode is also possible, again limited by the DMI when combined with other PCH-supported devices. All of these limitations are traceable back to Intel; there's little MSI can do about them.

Two USB 3.0 front-panel headers are lined up along the front edge, one facing upward above all the cards and the other facing forward behind the top card. Of course, you could call these USB 3.1 Gen 1 if you preferred, but their bandwidth is still limited to USB 3.0’s 5 Gb/s.

Referred to as Game Boost, the red button in the Z170A Gaming 7’s lower-front corner includes a dial with eight different overclocks. By skipping a few numbers, MSI is able to tell people that these go to 11. Presets range from 4.3GHz at 1.25V (level 1) to 5GHz at 1.45V (level 11) skipping levels 3, 5, 7 and 9. Other on-board controls include the Flashback button for USB firmware updates, a “Slow Mode” switch to drop the CPU to a more stable frequency when using LN2 (to avoid “cold bug” conditions under light load) and a switch to enable MSI’s “Gaming Hotkey” keyboard mapping function.

MSI slides its front-panel audio connector forward slightly from the bottom-rear corner, but probably not enough to avoid the rare (but dreaded) problem of poorly designed case cables. The company also adds labeling stickers and a thicker case badge to a Z170A Gaming M7 installation kit that has but four SATA cables and a flexible SLI bridge.

Software And Firmware

MSI Command Center still provides software overclocking, power/fan controls and even helped us determine the actual BCLK limit of firmware (since the firmware would allow nonfunctional BCLK settings). Command Center and firmware overclock settings are linked.

Command Center’s advanced voltage menu offers a surprise setting in CPU Ring Ratio, which is not a voltage. Core Voltage is confirmed through CPU-Z.

MSI fixed its DRAM settings menu, though this particular setting has a troublesome history. When it doesn’t work, it causes the program to crash.

The “Real-time Onboard Temperature Sensors Control” menu adds an interesting look to system stats.

MSI Gaming App includes a quick overclocking utility (4.3 GHz at 1.25V), “Eye Rest” monitor brightness and contrast settings, keyboard/mouse settings and on-board LED controls that include the network port's LED.

Live Update 6 now provides a save path pop-up for users who want to save the update files for later installation.

Nahimic Audio is the solution MSI chose (instead of Creative’s) to provide additional audio control. Settings include noise reduction and surround virtualization. MSI was so pleased with the results that it did a demo at Computex.

MSI has extended its Premium upgrade for XSplit Gamecaster to a full year.

MSI Click BIOS 5 opens up to a simplified menu containing links to a few basic controls, XMP mode, monitoring and firmware flashing utilities.

Our Core i7-6700K topped out at 4545MHz on the Z170A Gaming M7 with a 1.30V setting. Those last 55MHz were most likely unavailable as the result of voltage sag under high CPU load, as none of the firmware’s compensation controls were sufficient. Then again, when’s the last time anyone needed 55 more megahertz?

Our DDR4-3600 memory kit was also down, landing at a maximum stable data rate just below 3200 MT/s. The Z170A Gaming M7 required a 1.34V setting to stay slightly above 1.35 measured volts. We anticipate improvements through firmware updates.

The Z170A Gaming M7 includes more memory timings than we need and, like many competitors, can make automatic adjustments that are somewhat suitable to custom memory overclocks.

We couldn’t find an appropriate VDroop Compensation setting for our CPU, but perhaps CPU-Z was just misreading the voltage level. Nothing we tried here worked better than the “Auto” setting at getting our CPU passed its 4545MHz barrier.

The Z170A Gaming M7 has five different user-configurable fan maps for its five fan headers, and also includes a “Board Explorer” map to show where devices have been detected.

Supermicro C7Z170-SQ

Supermicro is banking on its reputation as a server component manufacturer in its “Server Quality” C7Z170-SQ. The board features a single USB 3.1 Type C port on the I/O panel, in addition to two USB 3.0 ports, two USB 2.0 ports, a PS/2 port, HDMI, DisplayPort, DVI-D, analog/digital audio and a gigabit Ethernet port.

Like its competitors, Supermicro’s C7Z170-SQ splits off eight of the top slot’s sixteen PCIe 3.0 lanes and sends them to the second x16-length slot whenever a card is detected. Also like its competition, the C7Z170-SQ has a third x16-length slot with only four lanes. But slot similarities end there, as the third slot borrows four lanes from the second. That means your options across the red slots are x16-x0-x0, x8-x8-x0, or x8-x4-x4. This could be a problem for anyone attempting to run two cards in SLI and a third card in the third red slot, since SLI is barred from functioning on four-lane slots.

Supermicro also feed four lanes from the Z170 PCH into the x4 slot between the second and third graphics slots. Anyone who wants to run SLI on the first and second red slots will likely find that the cooler of their second graphics card blocks access to the only 4-lane slot fed by the PCH. The two other x4-length slots are actually only wired as x1, and all three x4-style slots have open ends to support longer cards.

The top two x16-length slots are also closer together than competitors, at two spaces. That means you won’t be able to use two extra-thick graphics cards in SLI. Ordinary double-slot cards work, and we recommend that these have blower-style coolers when spaced this close together.

One of the benefits of moving the middle graphics slot up one space is that the lower slot can also be moved up a space, allowing access to the USB 3.0 header on the bottom of the board when double-slot graphics cards are used. Even if that header does get covered by a card, another USB 3.0 front-panel header can be found on the C7Z170-SQ’s front edge, facing forward.

A PCIe 3.0 x4 M.2 slot rests above the top PCIe slot to ease module access on a complete system. It’s not SATA-compatible.

Three buttons near the top-front corner ease use outside the case, and include CLR_CMOS, power and BIOS restore functionality (which instructs the board to read and load firmware from a USB flash drive).

As with competing models, the C7Z170-SQ’s infrequently encountered but hardest to overcome layout limitation is probably its rear-bottom corner front-panel audio header. Those few cases with cables too short to reach this location should be fixed by the case’s manufacturer.

On one hand, Supermicro shipped our C7Z170-SQ sample without its cables or CD. On the other, the instructions sheet says that these are the only things missing from the photo. Supermicro says it provides six SATA cables, which is two more than the competition.

Firmware

The version of C7Z170-SQ firmware we used was a little difficult to work with, partly because the board doesn’t make many automatic decisions on the user’s behalf and partly because the factory overclock settings don't allow further customization. For example, the factory 4GHz O/C group set the CPU core voltage to 1.40 to 1.50 volts, which caused an overcurrent alarm when loaded with Prime95. That voltage couldn’t be lowered without manually “un-setting” the 4GHz overclock. Other factory overclocks used even more voltage, up to 1.7V at 5GHz! Supermicro created a new firmware to address that, but we manually worked out our own settings before it arrived.

After taking some screen shots of the 5GHz overclock as a reminder of all the things Supermicro’s overclocking team changed, we were able to manually configure a 4.6GHz overclock at 1.3V CPU core. The key setting, it turns out, was found under the Core/IA VR Domain submenu, where VR Current Limit was increased to 1022 and VR Voltage to 7999, whatever that means. Without those changes, any increase in core voltage would cause the board to throttle the CPU to 3.6GHz under eight threads of Prime95.

The C7Z170-SQ misconfigured our memory’s XMP-3600 profile with several timings set to 0, and again would not allow alterations once XMP was enabled. The only way to achieve manual memory configuration was in “full” manual mode. Timings would remain from the previous mode, but with XMP misconfigured to the point that those timings wouldn’t work at any data rate, the only option was to start with baseline DDR4-2133 C15 timings.

After determining that any manual timing changes force all timings to manual configuration, and that any manual timing change (even from 15-15-15-35 to 17-18-18-36) would result in the board not rebooting at any memory data rate, the only option remaining was to increase data rate and voltage from stock settings without altering the original 15-15-15-35 settings.

Amazingly, the board reached a 3051 MT/s data rate at CAS 15. That kind of hardware stability means the C7Z170-SQ has lots of room to push impressive data rates as soon as Supermicro’s engineers configure its firmware for better memory kits.

Benchmarks, Power, Heat And Efficiency

We’re not able to use our reference test system’s LGA 2011-v3 processor or its motherboard to test LGA 1151 platforms, nor could we expect much memory overclocking from its DDR4-2400 quad-channel kit. But the rest of the system remains valid for this platform’s benchmarking and overclocking needs.

Test System Configuration

Synthetic Benchmarks

3DMark and PCMark show little performance difference between the three LGA 1151 motherboards in today’s test, but the Z170 Extreme6 does have a small glitch in Encoding/Decoding turnaround from Sandra Cryptography.

That small blip in the Cryptography module is caused by a small deficit in memory bandwidth. Perhaps ASRock was shooting for stability? Overclocking will help to determine that.

Gaming Benchmarks

All three boards look similar in games until we get to Far Cry 3. Several retests did not reveal the cause for performance differentiation in this one title, but the difference was small enough to negate the need for lengthier investigations. It is, after all, just one of the four games tested, and synthetics didn’t reveal a reason for the lowest-bandwidth board to have a 3D performance advantage.

Non-Gaming Applications

Less time means more performance in timed benchmarks. The Z170A Gaming M7 takes a small lead in the A/V encoding suite, but nowhere else, and by not enough to make a noticeable difference overall.

Power, Heat, And Efficiency

The C7Z170-SQ doesn’t have a bunch of added controllers, and it also has the lowest maximum and minimum power readings. The Z170 Extreme6 isn’t far behind, while the better-equipped Z170A Gaming M7’s extra components consume a little extra energy.

The result is a 2.5% efficiency lead against the average of all three motherboards for SuperMicro’s C7Z170-SQ. The Z170 Extreme6 takes second place, while the Z170A Gaming M7 is the part that pulls the entire class average for efficiency in the wrong direction.

Overclocking, Value And Final Analysis

Certain settings, such as multipliers beyond 83x on Skylake, can’t be set in hardware even when available in firmware. Other settings, such as 0.0 volts, won’t give a component any energy to operate. That’s why we take the range of overclock settings for each board with a grain of salt.

Overclocking Frequency & Voltage Settings

Several sources are reporting “safe” voltage limits for Intel’s Skylake processors from 1.40 to 1.45V, yet I personally have a tough time believing the company's engineers have improved voltage handling to a great extent in a single generation. Lacking the data to show how many months of reliability I could expect under 100% load at those high voltage levels, I stuck with 1.30V

ASRock and Supermicro achieved the top CPU overclocks, though only ASRock was prepared to use our DDR4-3600 memory samples. MSI is somewhat aggressive at firmware development, so we expect the race to tighten as we move on to review more mature products.

Then again, some motherboards use loose tertiary timings to increase memory overclock capability, losing performance in the process. Supermicro appears to lead the DDR4-2933 race only because its board couldn’t adjust timings automatically to increase stability. ASRock seems to lose this race while also leading the DRAM overclocking competition. And MSI’s middle position doesn’t look so bad when its two competitors swap places. Until someone does something extraordinary with their memory configurations, the rightful overclocking crown remains limited to CPU core clock where ASRock and Supermicro are tied.

A performance per dollar chart like the one above doesn’t compensate for the added cost of connectors, controllers and switches needed to turn a basic platform into a high-end motherboard. So MSI’s loaded-up $229 board looks like a modest value compared to ASRock’s more basic part. The question of whether the Z170A Gaming M7’s upgraded Killer e2400 network controller is worth more than the Z170 Extreme6’s extra SATA controller is probably best left to you and your needs. But MSI is very keen on its added Nahimic audio software, extra M.2 interface and one-year premium service upgrade for XSplit Gamecaster. The company's motherboard software is also far more advanced than ASRock’s, and the ASIC required for BIOS Flashback must be worth something. If you’re overclocking with LN2, the Slow Mode switch is invaluable as well. But are all of those differentiators worth $40?

Yes, they probably are. Of course, giving MSI a $40 credit without taking away some points for things that ASRock does right, such as its dual socket-mounted firmware and selection jumper, probably wouldn’t be right. And ASRock did win the overclocking competition.

So who gets the prize? ASRock might, except that I can't really give an award based on overclocking determined through beta firmware. MSI comes up next, except that it probably would have needed a little bit of that beta firmware treatment to overcome its slight overclocking deficit. So at this point, I’m going to reserve any awards for future products and use these $190, $210 and $230 boards as my baseline for comparing the improvements of our next round-up.

ASRock Z170 Extreme6

MSI Z170A Gaming M7

SuperMicro C7Z170-SQ

Update 8-07-2015

ASRock informed us ahead of time that its Z170 Extreme6 would be $180, but the cheapest launch-day price we found was $190 at Newegg. That vender has since lowered the price, as promised, to $180. This cost savings makes the Z170 Extreme6 an even better bargain for value seekers, particularly those who want to heavily overclock their memory.

MORE: Best MotherboardsMORE: 
How To Choose A Motherboard: A Guide For BeginnersMORE: All Motherboard Articles

MORE: Latest Motherboard NewsMORE: Motherboards in the Forums

Thomas Soderstrom is a Senior Staff Editor at Tom's Hardware, covering Cases, Cooling, Memory and Motherboards. Follow him on Twitter.

Follow Tom's Hardware on Twitter, Facebook and Google+.

More Features, Lower Price?

Remember when Antec’s lowest-cost case had very few features but was built like a tank? Builders continued ordering the old Three Hundred for several years past its expiration date, without so much as a USB 3.0 update, even as Antec kept trying to replace it with newer, more enthusiast-oriented designs. A quick check online suggests that the Three Hundred is finally dead. Long live the P70?

MORE: How To Build A PC: From Component Selection To Installation
MORE: All Case Articles
MORE: Cases in the News
MORE: Cases in the Forums

Exterior

Antec's replacement takes the identically-priced Three Hundred's minimalist appeal to a new level by hiding air inlets on the side of the face panel. Those side vents eliminate the direct noise path found on mesh-front cases. They also prevent “blinging out” the front with lighted fans. 

The P70 still appears to have three 5.25” bays. But unlike the Three Hundred, the P70’s lower external bay is permanently adapted to 3.5” drives.

This case also adds a pair of grommet-covered holes to ease external liquid cooler installation, but is still limited by only exposing seven expansion slots. The lack of an eighth prevents you from installing a graphics card with a double-width bracket in your ATX motherboard's bottom slot. All seven slot covers are non-replaceable break-outs, and the installation kit doesn’t include any replacements. 

The rear panel also features a row of vents next to the graphics cards, a 120mm exhaust fan and a latch for removing the slide-in top-fan cover. The fan grill back there is drilled to accept 80 and 92mm fans. 

Different from Antec’s published specs, two USB 3.0 ports and two audio jacks comprise the entire set of top-panel connectors. All of these feature rubber covers to prevent dust intrusion, though we know most enthusiasts will probably remove and lose them. Behind the ports and power/reset buttons, a trio of three-position switches control top, front and rear fans at two different speeds. 

The P70 front panel supports two fans, but both must be installed between the snap-away plastic and steel portions. A removable dust filter is also accessible only with the entire face plate removed. 

The power supply air inlet has a filter that must be bowed away from a barb on the metal chassis to remove. A screw and spacer act as a forward stop, preventin the filter from sliding in too far. 

Top fans are properly spaced for a radiator, but the P70 doesn’t have room between the panel and motherboard to install an internal liquid cooling radiator. Enthusiasts who opt for closed-loop coolers may instead use a 1x 120mm unit on the rear fan.

Building Inside The Antec P70

The P70 doesn’t have internal grommets. But cable passage holes are rolled over for a burr-free edge. Antec adds a couple of extra holes to clean up the installation of microATX boards. It also adds a lip at the front edge of the motherboard tray 9.75” from the back, though. Even slightly oversized boards won’t fit. 

Long graphics cards (up to 15.1”) fit into the P70’s top four slots; medium-length cards (up to 10.8”) drop into the bottom two. The stepped-in 2.5” SSD bay atop the 3.5” cage gives builders an extra inch of space (up to 11.8”) in front of the fifth slot. 

That gives you one 2.5” and four 3.5” internal bays, though the external 3.5" bay's integrated mounting bracket is also designed to attach hard drives. Screws for installing optical drives are mounted to sliding pegs that rattle around as the case is moved. 

The P70 only has a pair of USB 3.0 ports, so there's just one corresponding USB cable. The front-panel LED cable is split so that it supports both Asus (three-pin spacing) motherboards and everyone else (two-pin spacing). 

Surprisingly, the P70 has a cover plate for top-panel fan mounts, in case you want to move those fans to the front and keep dust from settling in through the top. It also comes with four sets of 3.5” hard drive rails, a set of twist ties for organizing cables, 16 foam washers for dampening screws and a screw pack with a bundled motherboard speaker. 

Side panel bulges are required since the P70 has very little space for cables. Builders must lay cables out carefully to minimize crossing. Otherwise, those thin side panels can easily be warped. 

The SSD bay requires straight cable connectors as well, which could be a problem on motherboards that only include right-angle cables. We reduced the number of power cables in our installation by using a four-pin drive cable and a four-pin-to-SATA power adapter. 

The lack of space for slightly oversized motherboards like our X99S XPower AC forced us to use the smaller X99S Gaming 7. This is starting to look like bothersome trend with mainstream cases. 

The P70’s power LED was bright enough to “blow out” the images we captured of it. We spliced two of them together rather than track down a bunch of extra foreground lights to minimize the effect. To the left of the main light row, two more lights indicate drive activity.

How We Test Cases

Test System Components

Settings

Drivers

To facilitate identical cooling on differently-sized motherboards, we're downsizing from Noctua’s huge NH-D15 to its NH-U12S. Though the smaller dimensions could solve fitment issues with some hardware combinations, cooling our overclocked Core i7-5930K is more challenging for its single-tower sink and one fan.

We’ve also transitioned from a noisy blower-style graphics cooler to an axial fan model from Gigabyte. The GV-N970G1 Gaming-4GD keeps its GPU exceptionally cool at reduced noise, while dumping its heat directly into the case.

Power comes from the 80 PLUS Platinum-rated Dark Power Pro 10 850W by be quiet!

Benchmark Suite

Designed by committee, our new test platform runs hot and quiet, negating the dramatic performance differences its predecessor was designed to produce.

Noise is measured .5m from the case’s front corner, on the side that opens. The numbers are corrected to the 1m industry standard used by many loudspeaker and fan manufacturers by subtracting six decibels.

Comparison Cases

Test Results And Analysis

Today’s test uses the same hardware and settings as the previous be quiet! and Supermicro case reviews, allowing us to produce comparable results without retesting previous samples.

Do those temperatures seem a little high? Generating test results on the P70 without encountering host processor thermal throttling was accomplished only after raising the CPU’s threshold (in UEFI) to 115 °C and lowering the room temperature to a chilly 15°. Though we might have been able to get the CPU temperature down by 10° by using a huge cooler, the P70 appears to be designed for lesser hardware. Perhaps a Core i5 at a similar clock rate and voltage?

Super-thin side panels do little to isolate internal component noises from the outside of the case. Even the Supermicro Gaming S5 with its vented side-panels is slightly quieter.

Hot and noisy are not good adjectives in a cooling-to-noise comparison, though the P70 has a lower price to help excuse its performance deficit. Cheap system builders don’t want canned heat, but cheaper processors usually have less heat to manage.

The point is seen in our charts, where even the combination of poor cooling and noise control isn’t enough to prevent a value win. We should probably recommend the P70 to someone as a result. But we still aren't quite sure who this case is built to attract. Even as the P70 occasionally dips below $60, we wonder whether we should do an exhaustive search for a higher-quality product at a similar price or drop an extra $20 on a true budget-oriented enthusiast chassis. Antec sells those too. Even its venerable Nine Hundred is available for less than $100. Given what we’ve seen here, we’d probably spend more on something better.

Conclusion

MORE: How To Build A PC: From Component Selection To Installation
MORE: Cases in the News
MORE: Cases in the Forums

Introduction

Do you want a big, heavy, sturdy case that can hold a stack of hard drives from its top to bottom? That’s the kind of chassis most of us associate with the Supermicro brand. Would you prefer something cheaper and lighter? Until recently, most of those cases came from companies that didn’t compete in any of the markets Supermicro served. Keen to that opportunity, Supermicro began inching into mainstream PC markets with motherboards, and is now expanding to include cases as well.

MORE: How To Build A PC: From Component Selection To Installation
MORE: All Case Articles
MORE: Cases in the News
MORE: Cases in the Forums

Exterior Tour

A splash of color and the sheen of brushed aluminum separates its Gaming S5 from Supermicro’s traditional products. You're also looking at a sub-20” depth measurement. Given the company's traditions, we were expecting at least two feet. This might be the first Supermicro desktop component we’ve tested that actually fits atop most desks. Of course, you could also shove it under the desk. But then I’d wonder why you want to hide it. Either way, the pair of USB 3.0 ports, headphone and microphone jacks are lined up along the top of the front panel to ease access from either location.

A pop-out panel at the bottom of the front face features mesh backing to catch large dust particles. Two 120mm intake fans behind the filter can be upgraded to your choice of 120 or 140mm replacements.

Supermicro includes (but does not install) the two grommets for its rear-panel coolant line passages, instead leaving round knock-outs in their place. Extra space for radiator mounting is highlighted by a row of vents above the motherboard’s I/O panel, leaving the presence of only seven expansion slots as the S5’s most obvious externally-visible limitation. That is to say it won't take a full complement of graphics cards; an eighth slot is required to place a double-space graphics card into an ATX motherboard’s bottom slot.

A glance at the dimensions uncovers a few additional restrictions, such as the 11.2” maximum graphics card length in some slots when all drive cages are installed. But a look inside will reveal configuration options for users of longer cards. So, let’s take a peek!

Interior Tour And Building

The Gaming S5 officially supports motherboards up to 10” wide behind its three modular drive cages. Grommets dress up cable passage holes above, in front of and below the motherboard, and an extra hole behind an ATX-sized board allows cleaner cable management on microATX boards.

By default, the 2.5” drive cage is centered between 3.5” cages. Though that fact doesn’t fit neatly into the previous page's dimensions table, it allows the placement of expansion cards up to 12.6”-long in slots two through five. That is, until you move things around.

Covered by a magnetically-attached mesh sheet, the S5's top panel supports multiple spacing options for up to three 140mm or 120mm fans. Though you won’t find space for the end caps of radiators with three 140mm fans, a 3x120mm configuration fits down the center thanks partly to the top panel’s two inches of motherboard clearance. Thicker radiators can be offset to the left side, though upper drive bays limit this location to a 2x120mm length.

The same type of mesh that covers top-panel vents also filters dust at the power supply and lower fan inlets. Users with standard-sized power supplies (PS2 form factor) could potentially put a 2x120mm radiator here, though most high-capacity power units are too long to allow this.

Removing a drive cage extends maximum card length to 16.5”. A modular design allows these to be stacked any way you wish. Using slide tabs, a single cage can be hung from the top or secured to the base, and a screw at the front of each cage prevents it from sliding out.

Slides for the lowest drive cage engage a screw-in tray. Removing the tray provides access to the forward bottom-panel fan mount, and is required for fitment of a front-panel radiator. Users who don’t need any of the drive cages and can use a relatively short power supply will find room for up to three radiators!

Front-panel bay adapters make it possible for some builds, especially gaming machines, to get by without the drive cages. Mounting patterns for 3.5” and 2.5” drives are found behind the black-anodized brushed-aluminum face plate.

Screw-free mounting pins are found on both 3.5” and 2.5” trays, and additional 2.5” drives can be screwed onto 3.5” trays.

The Gaming S5 has enough space behind its motherboard tray to accommodate an ATX/EPS main power lead, along with other cables. But its access holes are very small. We found it necessary to remove the grommet if we wanted to route a 24-pin connector behind the tray. Fortunately, the grommet can be stretched around that connector once it's removed.

In addition to the expected screw kit and grommets to replace the two rear-panel knock-outs, Supermicro adds a front-panel header adapter for motherboards from other vendors. That's because the case’s stock cable is designed exclusively for Supermicro platforms.

Most motherboards (apart from Asus and Supermicro) use the Intel-specified nine-pin front-panel LED/switch header, which places power LED wires adjacently. Carried over from the old AT form factor and retained by Asus, Supermicro’s adapter spaces these pins three positions apart. Splitting the connector allows it to fit both standard and Asus front-panel headers.

An ATX motherboard (12” x 9.6”) fits nicely, but oversized platforms like the 12” x 10.7” MSI X99S XPower AC I like to use would block access to cable passages along the front edge. That could be a big deal for many high-end builders, since a lot of enthusiast-oriented motherboards are similarly oversized.

The finished build looks clean and stylish. But the side-panel vent is a little worrisome concerning noise. Let’s find out if Gaming S5 results can overcome those worries!

How We Test Cases

The system we used in this article only deviates from our official 2015 Reference article in motherboard. We had to swap out the reference system's 10.5"-deep MSI X99S XPower AC for a standard ATX (9.6") model: the X99S Gaming 7.

Obviously, our Lian-Li PC-T80 open bench chassis sits this one out.

Test System Components

Settings

Drivers

To facilitate identical cooling on differently-sized motherboards, we're downsizing from Noctua’s huge NH-D15 to its NH-U12S. Though the smaller dimensions could solve fitment issues with some hardware combinations, cooling our overclocked Core i7-5930K is more challenging for its single-tower sink and one fan.

We’ve also transitioned from a noisy blower-style graphics cooler to an axial fan model from Gigabyte. The GV-N970G1 Gaming-4GD keeps its GPU exceptionally cool at reduced noise, while dumping its heat directly into the case.

Power comes from the 80 PLUS Platinum-rated Dark Power Pro 10 850W by be quiet!

Benchmark Suite

Designed by committee, our new test platform runs hot and quiet, negating the dramatic performance differences its predecessor was designed to produce.

Noise is measured .5m from the case’s front corner, on the side that opens. The numbers are corrected to the 1m industry standard used by many loudspeaker and fan manufacturers by subtracting six decibels.

Comparison Cases

Test Results

Even though the newer test hardware produces closer results, we can clearly see that the mid-sized Gaming S5 runs cooler than the much larger Corsair 760T.

Perhaps that extra coolness is due to the side vent, which allows heat to escape from an internally-vented graphics card? Unfortunately, those same holes in the side panel allow noise to escape as well.

As a result of the extra noise (which admittedly isn't much, given the quiet cooling used), the Gaming S5 falls 2.5% below the average of all three cases in acoustic efficiency. That is to say, it has a marginally worse cooling-to-noise ratio.

On the other hand, the Gaming S5 also has a $94 MSRP. We normally use prices only from popular etailers to determine street pricing, but are sticking to MSRP for the Gaming S5’s calculation because it’s not readily available. We found various small sellers offering this Supermicro model-number CSE-GS50-000R for $89 to $136.

Conclusion

Supermicro’s Gaming S5 looks like a great value, but that’s partly because it’s the first sub-$100 case I’ve tested with the new hardware. As I wait for the next mid-market case to come in, let’s consider a few Gaming S5 specifics.

First of all, the S5 is very light and built with extremely thin steel. Supermicro does an admirable job of bracing it internally with rolled edges and a complex drive cage system, but side panel flex can still be a big deal. It’s particularly hard to put back together with that space behind the motherboard tray stuffed with cables. You'll need several hands to line up its slide tabs when cables are pushing against such a springy panel. We saw this issue addressed by adding a broadly-boxed edge on the Silent Base 800 side panels from be quiet! Then again, that case was 50% more costly and still made out of similar-gauge material.

The Gaming S5 also lacks space for slightly oversized motherboards, which are popular at the high-end of most enthusiast motherboard manufacturers. Maybe you won’t put a $400 board in a sub-$100 case, but I’m sure someone will try.

Perhaps the least-forgivable problem is that access holes are too small to pass a 24-pin cable, at least without removing the grommets. Then again, once you get the cable through, you’ll probably leave it there for a long time. It’s just the kind of thing that would have me pulling a Gibbs as the leader of a design department.

On the other hand, the S5 has several excellent design attributes that are sure to please most builders, such as the triple drive cages that can be independently installed or removed, and space for up to three double-fan radiators. Or even a triple-fan, with a bit of effort. The black-anodized brushed aluminum also looks far more convincing than the brushed-texture plastic of many competitors.

Thomas Soderstrom is a Senior Staff Editor at Tom's Hardware, covering Cases, Cooling, Memory and Motherboards. Follow him on Twitter @hardware_tom.

Follow us on Twitter @tomshardware, on Facebook and on Google+.

MORE: How To Build A PC: From Component Selection To Installation
MORE: Cases in the News
MORE: Cases in the Forums

Reliable Overclocking?

Famed for its reliability in the server market, Supermicro was once known to enthusiasts only for its cases. The company finally decided to dip a toe into the overclocking market with the C7Z87-OCE motherboard. But it wasn’t until Intel's Z97 update that Tom’s Hardware finally felt the teething pains of a company that was most experienced building business-oriented hardware. Today’s C7X99-OCE brings over two years of overclocking-related firmware advancements to users who want the added cores or PCIe 3.0 lanes of Haswell-E via LGA 2011-v3.

The C7X99-OCE is even rated for four-way SLI, though the dual-slot brackets and coolers of most compatible cards makes it unlikely that enthusiasts will try squeezing a third card into that single space before the fourth slot. You’d also lose front-panel USB 3.0, since the solitary header is located along the bottom edge. Heck, you might even lose the use of that header with a third card, depending on whether your double-space cooler is more than 8.4” long. Before we dig deeper into the connectors, here’s how it compares to the rest of our ~$300 boards:

More C7X99-OCE Details

The C7X99-OCE’s four x16-length PCIe lanes are hard-wired with x16-x8-x8-x8 pathways, so you’ll never get true x16 mode for three-way SLI. That’s not a worry for us, since we haven’t tested a card yet that needs more than eight lanes of PCIe 3.0 to reach optimal performance. On the other hand, a fixed x8 slot might bother a few purists.

The hard-wired slot arrangement could be a little more troublesome to owners of 28-lane (Core i7-5820K) processors, since the reduced lane count is divided into x16-x0-x8-x0 pathways.

And the documentation couldn’t be less clear. Even after figuring out that engineers numbered these slots from left to right (rackmount orientation) rather than top to bottom (for tower users), apparently conflicting information on the manual and website forced me to test every possible slot configuration with two Radeon R9 290X graphics cards and two processors (Core i7-5960X and -5820K) to confirm that the things I wasn’t seeing weren’t there.

Two additional card slots house PCIe 2.0 x1 interfaces within x4 connectors.

Builders who are able to avoid covering the bottom edge will have access to a row of overclocking buttons, beginning with the baseline “home” button, progressing to the motorcycle-themed OC1 button (rated at 15% O/C), the F1 race car OC2 (rated at 20-25% O/C), a fighter jet-looking OC3 (user-programmable) button and a Memory O/C button. Deploying OC1 gave us 4GHz at 1.4V CPU core and cache, OC2 facilitated 4.3GHz at 1.4V CPU core and cache and Memory O/C triggered our memory’s DDR4-3000 XMP profile. We’ll discuss that on the next page.

Additional buttons include a CLR-CMOS button just above the second x16 slot on the C7X99-OCE’s rear edge and a BIOS Recovery button next to its two-digit status coded display.

You might have noticed that the C7X99-OCE doesn’t have any USB 2.0 front-panel headers, or even rear-panel ports. Lined up along the bottom edge, data headers include only a nine-pin serial port in the back corner, a proprietary interface for Supermicro’s OC Front Panel bay adapter toward the middle, a TPM header, front-panel power/reset/LED connections and USB 3.0. OEM customers can have four of the board’s eight USB 3.0 headers upgraded to USB 3.1 for an added fee.

The front-panel audio connector is found next to I/O panel jacks, easing its implementation with short cables. Some builders might complain that they can’t hide a cable that runs across the motherboard’s surface, but we’ve seen far too many cases with cables that can’t reach the traditional bottom-rear corner location (where Supermicro places the C7X99-OCE’s serial port).

Stripping a motherboard down to its essentials leave little room for error when it comes to component placement. The potentially-problematic front-panel USB 3.0 header is far enough from the third graphics card to allow most long graphics coolers to fit, and all of the SATA headers face forward to avoid conflict with extra-long cards. And while five isn’t a generous number for four-pin fan headers, at least these are spaced properly for two front case fans, a rear case fan and front/rear CPU cooler fans.

Supermicro didn’t include the C7X99-OCE’s installation kit in our package, though a quick look at one of our preferred sellers shows that it includes six SATA cables and a single SLI bridge. Supermicro instead added its OC Front Panel accessory, which duplicates most of the board’s overclocking-centric buttons to a case’s 5.25” bay. This device is not specific to our board, however, as there are no headers on the C7X99-OCE to connect its USB 2.0 port.

C7X99-OCE Firmware

Supermicro’s firmware GUI resembles a touchscreen interface, placing most user-configurable options under its Overclocking menu. This includes most memory controls, since the memory menu includes only basic settings.

The CPU overclocking menu includes several pre-defined overclocks, ranging from 4.2GHz at 1.40 Vcore/Vcache, to 4.8GHz at 1.5 Vcore/Vcache. Our CPU is far too voltage-sensitive to work with any of those settings though; all of them cause near-instantaneous thermal throttling no matter how large our cooler.

This CPU reached 4376MHz at 102MHz BCLK, a 43x CPU core to BLCK multiplier and 1.28V. Adjusting BCLK and voltage isn’t like any other board we have in-house, as typed-in settings require using the backspace key to remove current values.

This configuration also encountered an issue with “CPU Short Duration Power Limit Switch” set to “Disabled”, where it would mysteriously throttle down the CPU clock quickly after initiating a high-current load. A 512W setting proved adequate, even though the processor never needed more than half as much.

SVID Override Voltage is a setting to watch out for. The board automatically set +600mV when initiating DDR4-3000 XMP values, and that setting appeared to cause random resets.

Manually configuring CPU input voltage to 1.90V (rather than relying on automatic voltage control and offset) appeared to provide the best stability with our hardware.

Even though the C7X99-OCE supports a wide enough range of memory multipliers to reach DDR4-4000 at stock BCLK, the board still chose a 125MHz BCLK for our DDR4-3000 data rate. That might not have been a problem, except we weren’t able to find an effective setting to reduce the CPU multiplier. Disabling Turbo Boost meant nothing to the board, setting a lower non-Turbo Boost ratio didn’t help under load and any Turbo Boost ratio below 35x was ignored. Settings that involved this memory’s XMP profile required a 4.38GHz CPU core, and the only way the memory could even be tested at DDR4-3025 was to increase core voltage and allow the CPU to throttle back.

I tried using manual configuration with XTU timings to reach DDR4-3000 without CPU overclocking, but the board refused to boot. Furthermore, the DRAM voltage had to be manually configured, as the board didn’t configure its XMP-rated voltage. The board’s 20mV default DRAM overvoltage paired with 50mV settings and our voltage-sensitive CPU to require the 1.30V setting, rather than 1.35V, to reach a stable 1.32V rather than an unstable 1.37V.

Probably ideal for owners of a perfectly-paired memory and CPU configuration, primary, secondary and tertiary timings are all adjustable.

Results: Benchmarks, Efficiency And Overclocking

Hardware and software settings carried over from our initial round-up allow me to compare the performance of every tested X99 motherboard to the most recent products. Charts include the three most similarly-priced competitors.

3DMark And PCMark

We haven’t seen significant attempts to game the review system in recent years, which would be indicated by noticeable increases in basic benchmark performance. That’s good, since those types of events previously triggered lengthy investigations into clock speed manipulation. Everyone can overclock, and overclockers prefer to use their own settings.

3DMark and PCMark show all four boards producing similar performance. No cheating here.

SiSoftware Sandra

Memory performance is one place motherboard manufacturers can still affect performance a little without overtly cheating, since certain advanced timings are based on other memory/motherboard factors and not programmed onto the module. Tighter timings tend to produce higher bandwidth scores, while looser timings lend themselves to additional stability. Supermicro appears to choose stability, as inferred by its slightly lower Sandra Bandwidth and Encoding/Decoding scores.

3D Games

The C7X99-OCE is also found near the middle of a tightly-grouped pack of gaming performance scores.

Encoding, Creativity, Productivity And File Compression

The C7X99-OCE retains mid-pack performance consistency throughout our application suites.

Encoding, Creativity, Productivity And File Compression

The C7X99-OCE consumes about as much power as its competitors, falling between the feature-packed Asus and feature-light MSI boards under full load. Temperatures are a little high, which could account for average power consumption on a board with fewer added-on controllers.

Our efficiency chart averages all nine previously-tested X99 motherboards, including microATX and low-cost models. The C7X99-OCE’s efficiency looks a little worse, even though the lower-cost boards it’s competing against have a similar feature count.

Overclocking

Supermicro endows the C7X99-OCE with a competitive range of clock settings, and even includes the non-validated memory ratios familiar primarily to owners of Asus motherboards.

My test CPU unfortunately didn’t reach the expected 4 to 4.4GHz, coming up just shy of target at 4376MHz using a 102MHz BCLK setting and 43x multiplier. A slight shortfall in DRAM overclocking occurred when attempts to lower the CPU multiplier were either ineffective or caused a boot failure, though users of slower RAM should experience a greater variety of workarounds. Higher memory multipliers didn’t work either, though manual timing configuration might have eventually solved the issue. After five days of trying various firmware configurations, the 0.025GHz shortfall in frequency and data rate seemed even less important.

Choosing DDR4-2666 rather than the default XMP-3000 setting might be an option, but it barely outperforms DDR4-2133 SPD values on the C7X99-OCE. This chart is primarily a test of how aggressively each board auto-configures secondary and tertiary timings in response to multiplier manipulation, and it appears Supermicro favors overclocking stability over performance superiority.

Final Analysis

Still famous for its server boards and heavy-duty cases, Supermicro’s consumer motherboards trail the competition when it comes to ease of overclocking. Targeting enthusiasts makes this an important factor, though it certainly won’t be as noteworthy to anyone who doesn't spend time tuning their hardware (or prefers a challenge overcoming performance obstacles).

We’re still using all nine boards to rate comparative value, which means that high-end contenders look somewhat worse than we might expect, while low-end boards fare better. To make this comparison more relevant, we need only look at the C7X99-OCE’s less than 1% value deficit compared to the Asus X99 Pro.

The C7X99-OCE supports four-way SLI, though good luck finding a single-slot card that'll let you use the third and fourth PCIe slots. On the other hand, Asus' X99 Pro supports three-way SLI, though adding an M.2 card breaks that. And the C7X99-OCE doesn’t even support M.2. Asus only wins this face-off if you need M.2 and don’t need three-way SLI. For most users, it’s a three-way SLI tie.

There’s always the trouble of processors with 28 lanes of third-gen PCIe. Right now, that includes the Core i7-5820K. Supermicro and Asus both lock the first slot into x16 mode, limiting how the remaining 12 lanes can be spread and limiting both boards to two-way SLI. Asus allows the remaining four PCIe 3.0 lanes to be used for its M.2 slot, but that only matters to storage enthusiasts buying PCIe-based M.2 drives. It’s a small win for Asus.

Both motherboards have two network controllers, but Asus’ second controller is a more costly wireless version. That would make the Asus motherboard worth more, except that some power users prefer two GbE wired connections.

Overall value considerations imply that value is in the eye of the beholder, yet Asus already lost the battle of features for your dollar to ASRock's better-equipped X99 Extreme6/ac. And both of those boards lost the SLI fight to MSI’s quad-SLI-spaced and properly-switched X99S MPower. That only leaves overclocking (which Asus again wins), additional components like USB 3.0 controllers (which all three competitors have) and the implementation of integrated features like USB 2.0 (where the C7X99-OCE doesn’t even have an internal header for the company’s own OC Front Panel device).

Still, we know there’s a market for this specific board, and it’s not just for fans of Supermicro’s legendary stability and/or service. I once spoke to a builder who asked, “why do we need all these extra ports and headers?” He added, “I want a clean board without all that stuff”. For them, there’s Supermicro’s C7X99-OCE.

Xeon E5-2600 v3 Platform Introduction

Today marks the launch of Intel's Xeon E5-2600 v3 processor family, based on the Haswell-EP design. We knew this day was coming, since the company already introduced its Haswell-E-based Core i7s. Of course, the Xeon family is Intel's mainstream server/workstation processor family, and the E5-2600 series is perhaps the highest volume line-up in the Xeon portfolio. It is also responsible for forcing AMD's Opteron 4000 and 6000 CPUs into relative submission. Now, the competition is refocusing efforts on low-end ARM-based processors.

The dual-socket server market is absolutely huge. So, any major technology refresh in the segment triggers billions of dollars in refresh purchases. HP already announced its new ProLiant Generation 9 servers and other vendors will roll out their own implementations starting today. Most server systems have a field life of three to five years. It follows, then, that Haswell-EP-based processors will replace platforms built on Nehalem-EP, Westmere-EP, and Sandy Bridge-EP. And unlike most desktop PCs, every dual-socket server can easily cost many thousand dollars.

As you are undoubtedly aware, there are three distinct lines under the Xeon banner. These E5s represent Intel's mid-range platform. The E3s more closely align with mainstream desktop core configurations, while the E7 tier is higher-end, scaling up to eight processors, many terabytes of system memory, and enabling RAS features for mission-critical applications. The E5 is a utility player of sorts, handling everything from heavily virtualized workloads to bare metal HPC applications. The "2" in the part number lets us know that we're looking at single- and dual-socket-capable parts. The "6" immediately following loses some of its meaning this time around. Previously, Sandy Bridge-EP- and Ivy Bridge-EP-based processors were also available as Xeon E5-2400s, which weren't as fully-featured. There is no Xeon E5-2400 v3 this time around, though. As of now, the E5s are 2600-series chips.

With Sandy Bridge-EP (Xeon E5-2600), we saw as many as eight cores manufactured using a 32 nm process. Ivy Bridge-EP (Xeon E5-2600 v2) benefited from a process shrink to 22 nm, enabling core counts as high as 12. Haswell-EP (Xeon E5-2600 v3) is being productized in configurations as wide as 18 cores. Each generation follows the core design and incorporates much of the technology that we see with the aligned consumer segment. That means, with Haswell-EP, voltage regulation circuitry moves on-package instead of residing on the motherboard. Another major change (already seen on the desktop) is Haswell-EP's LGA 2011-3 interface, which is not compatible with Sandy Bridge-EP, Ivy Bridge-EP, or the new Ivy Bright-EX's 2011-pin socket. The new interface facilitates DDR4 memory compatibility, delivering lower power, more density, and higher data rates than previous generations.

Here is a quick overview of the different model differences in the Intel Xeon E5-2600 v3 generation:

Clearly, the number of SKUs is massive. Intel tells us that three dies are used to create all of these different CPU models. Remember, many of the systems Haswell-EP will replace currently employ Westmere-EP, which allowed up to two sockets with six cores each. Common DDR3 data rates were 1066 and 1333 MT/s. Updating to Xeon E5-2600 v2 makes it possible to put two to three times as many cores into the same form factor and likely reduce power consumption at the same time.

Spanning four to 18 cores and up to 3.6 GHz base clock rates, Intel is enabling CPU models that are optimized for many different markets. Thermal design power ratings range from 55 to 145 W on the server side, and as high as 160 W for the Xeon E5-2687W v3 workstation part. That includes the fully integrated voltage regulator (FIVR) also seen on Intel's desktop-class Haswell processors.

One other note: this is the preliminary planned SKU composition. We know Intel is customizing processors for EMC, NetApp, and other large customers requiring specific feature sets. Those are generally not listed as public SKUs, though.

Meet Intel's Grantley Platform

In addition to its new Haswell-EP-based CPUs, there's a lot more to Intel's Grantley platform. We were given the following quick reference guide, which covers the basics:

There are a number of evolutionary changes to account for, but perhaps the biggest is Grantley's memory support. Four generations of server platforms dating back to Nehalem-EP utilized DDR3 RAM, and we've seen efforts to further tweak that standard for lower power use or greater density. Registered DDR4 DIMMs successfully achieve those improvements, additionally increasing throughput per channel.

Servers are often loaded up with RAM to handle more VMs or even to expand the space available for in-memory storage applications like memcached or redis. This typically requires more DIMMs per memory channel, imposing penalties on the peak data rate you're able to hit. DDR4 is designed to accommodate more DIMMs in a configuration without the performance penalty suffered by DDR3. And because it operates on a lower input voltage than even DDR3L, energy efficiency is built-in to the spec.

Of course, memory support is a product of the CPU's integrated memory controller. But not all system functions are built into Intel's processors yet. You still need a platform controller hub for a lot of the peripheral connectivity and I/O. The Wellsburg PCH, much like the already-reviewed X99 Express, exposes 10 SATA 6Gb/s ports. That's a significant upgrade to the Xeon E5-2600 v1 and v2 platform, where the focus was on adding optional SAS connectivity. Intel is clearly taking a different tact to coincide with the introduction of its NVMe-based SSDs. We're plenty happy with expanded SATA support, which is great for low-cost SSDs and traditional mechanical disks. High-performance storage is moving to the PCIe bus.

Other features include six USB 3.0 ports and eight second-gen USB connectors, useful for faster KVM cart access and accelerated boot from an internal VMware ESXi USB key installation. Several of the platforms we've seen in the lab are USB 3.0-only, in fact. That's a significant change from previous generations limited to USB 2.0.

The CPUs still enable 40 PCI Express 3.0 lanes, divisible into a number of different link configurations. This is a common feature on processors in the -EP range. With faster networking in this generation and a renewed focus on PCIe-based flash storage, all of that connectivity should go to good use.

Later in this article, I'll cover how power consumption and distribution change with Haswell-EP. The key is that, again, voltage regulation is on-package, and P-state control is more granular. As we saw on the desktop, this results in low idle power use. But unlike Haswell in its mainstream form, Haswell-EP packs up to 4.5x as many execution cores and more than five times the last-level cache. And in dual-CPU arrays, the effects of power savings are doubled per machine.

At least in my opinion, the most exciting platform change involves networking, including the 40 GbE Fortville controller...

Fortville: 40 GbE Ethernet For The Masses

Alongside the Grantley platform, Intel is introducing a new generation of Ethernet network adapters code-named Fortville. The controller is 40 Gb-capable, and it completely changes the game with regards to Intel's networking capabilities.

Given the massive performance increases enabled by today's systems, low-latency and high-bandwidth networking is essential. Grantley is built to consolidate an even greater number of virtual machines on a single server. Technologies like VMware vSAN utilize local storage to create distributed SANs for these virtual machines. Even trends like software-defined networking can benefit from higher port counts and greater networking performance. Fortville is Intel's solution.

There are three main configurations of the Fortville adapter: 2 x 40 GbE, 1 x 40 GbE, and 2 x 10 GbE. Compare that to the previous generation of Spring Fountain-based X520 adapters capable of up to 2 x 10 GbE. You quickly see the bandwidth "potential" goes up from 20 to 80 Gb/s on the Fortville X710 family.

While a 4x bandwidth increase sounds awesome, that ceiling is currently not possible to hit. Servers generally expose a bunch of eight-lane PCIe slots. Right now, the third-gen standard gives you a little less than 8 GB/s of throughput. Furthermore, there is always some overhead involved. So, hooked up to the Cray-Gnodal GS0018 (18 x 40 GbE) switch in the lab, we are seeing between 50 and 55 Gb/s of peak bandwidth. We dno't have enough data to publish formal numbers. However, there is definitely a bottleneck in play.

Still, in most deployments, the two QSFP ports will attach to different switches for failover. And there's plenty of headroom in the card to drive essentially a full 40 Gb connection, plus a solid amount through the second port.

Another aspect of 40 GbE-based products like the XL710 is that each QSFP port can utilize QSFP-to-4x SFP+ breakout cables. This allows for each XL710 card with dual QSFP ports to connect to up to 8x SFP+ 10 GbE devices. In theory, you can place eight of these dual-port cards in a server and then use that machine with 72 10 GbE network connections. The reasons not to are pretty obvious, but it's at least technically possible.

The other side of the Fortville story, aside from the controller's huge performance and density improvements, is power consumption. Fortville uses less power than the previous-gen X520 10 GbE adapters, both at idle and under load. The X520s had an 8.6 W TDP, while the XL710 generation is rated at 7 W. In a theoretical efficiency metric, Fortville delivers more than 3.5 times the throughput/watt compared to previous generation. That is a massive leap forward. Consistent with this, Fortville is rated at 3.6 W typical power consumption using two 40 GbE links, so 7 W TDP represents a lot of headroom.

At the end of the day, Intel's Fortville-based adapters should enable more bandwidth, lower latency, and higher port density, all at reduced power consumption.

How We Tested

Today's tests involve typical 1U server platforms. Supermicro sent along a new 1U SuperServer configured with two Intel E5-2690 v3 processors and 16 x 8 GB DDR4-2133 DIMMs from Samsung. We had a similar 1U Supermicro platform and pairs of Intel Xeon E5-2690 v1 and v2 processors to create a direct comparison. The Xeon E5-2690s are generally considered the higher-end of what ends up becoming mainstream. For example, companies like Amazon use the E5-2670 v1 and v2 quite extensively in their AWS EC2 compute platforms. The -2690 generally offers the same core count, just at a higher clock rate.

Intel also sent along a 2U "Wildcat Pass" server platform that was configured with two Xeon E5-2699 v3 samples and 8 x 16 GB registered DDR4 modules (with one DIMM per channel) and two SSD DC S3500 SSDs. The E5-2699 v3 is a massive processor. It wields a full 18 cores capable of addressing 36 threads through Hyper-Threading. Forty-five megabytes of shared L3 cache maintain 2.5 MB per core, and the whole configuration fits into a 145 W TDP.

Naturally, this is going to represent a lower-volume, high-dollar server. But it's going to illustrate the full potential of Haswell-EP, too. We're using the Wildcat Pass server as our control for Intel's newest architecture.

Meanwhile, a Lenovo RD640 2U server operates as our control for Sandy Bridge-EP and Ivy Bridge-EP. It leverages 8 x 16 GB of registered DDR3 memory, totaling 128 GB. We also dropped those SSD DC S3500s in there, too.

As we make our comparisons, keep a few points in mind. First, at the time of testing, DDR4 RDIMM pricing is absolutely obscene. Street prices are several times higher per gigabyte than DDR3. This will come down over time as manufacturing ramps up. But prohibitive expense did affect our ability to configure the servers with more than 128 GB.

We are focusing today's review on processor performance and power consumption. As a result, we are using the two SSD DC S3500s with 240 GB each in a RAID 1 array. We did have a stack of trusty SanDisk Lightning 400 GB SLC SSDs available. But neither of our test platforms came with SAS connectivity. Although there are plenty of add-in controllers that would have done the job, there is clearly a market shift happening away from such configurations. Sticking with SATA-based SSDs kept the storage subsystem's power consumption relatively low, while at the same time leaning on a fairly common arrangement in servers reliant on shared network storage.

Bear in mind also that we're using 1U and 2U enclosures, each with a single server inside. The Xeon E5 series is often found in high-density configurations with multiple nodes per 1U, 2U, or 4U chassis. For instance, the venerable Dell C6100, based on Nehalem-EP and Westmere-EP, was extremely popular with large Web 2.0 outfits like Facebook and Twitter. Many of those platforms have been replaced by OpenCompute versions, but we expect many non-traditional designs to be popular with the E5-2600 v3 generation, especially given its power characteristics.

Supermicro SYS-6018R-WTR

The main Xeon E5-2600 v3-based platform we've been using for testing is Supermicro's 1U SYS-6018R-WTR. It's largely an evolution of the SYS-6017R-WRF we already had in the lab. Although this is a 1U format, the server has redundancy built-in, and packs the space with features.

First, as a 1U chassis, there are only so many options for front-mounted storage. This particular chassis exposes four 3.5" front hot-swap bays. In our test system, bays one and two are populated with the Intel SSDs. There are standard LED indicators, as well as power and reset buttons. The rest of the chassis' face serves as a large vent, pulling air in and over the power-hungry components inside.

Redundant fans are responsible for this task. Essentially, two fans are spliced together. If one fails, the other continues to operate. In datacenters, emergency remote hands to replace a fan can cost $100. So, minimizing the need for urgent replacements saves money. Furthermore, with the redundant design, if a fan does fail, the system continues to receive some cooling.

In the same vein of redundancy, there are two 700 W 80 PLUS Platinum-rated PSUs in the rear of the chassis. Low-cost 1U designs, often sporting single Xeon E3s, are usually sold with one power supply to reduce costs. Higher-end servers like this one are meant to be fed with A+B power, and thus have the ability to weather a failure along one of the power delivery routes. That is to say each power supply is fully capable of keeping the server running independently.

The back of the chassis exposes fairly standard I/O. There are two built-in Ethernet ports and one IPMI/KVM-over-IP port for remote management. If you've ever experience a serious failure from a remote location and used KVM-over-IP to troubleshoot, you already know that this feature is awesome. Supermicro also enables four USB ports and a VGA output. Interestingly, this server does not have a dedicated serial port. You can always attach a USB-to-serial adapter if it's really needed.

Inside the server there are two expansion bays. For testing, we used one of the PCIe riser's slots for Supermicro's Fortville-based dual 40 GbE adapter.

Plastic guides ensure that air flows through the 1U heat sinks, RAM, and expansion cards in a focused fashion.

Our system came equipped with eight 8 GB Samsung DDR4-2133 ECC RDIMMs. When we received the test unit, these were very hard to purchase on the open market. You can see that the server has four memory slots on either side of each processor, totaling 16.

Linux-Bench Components And Test Setup

While Windows Server is still a popular platform, many Xeon E5-based servers are also going to run some flavor of Linux as an operating system. Server-oriented hardware also has relatively poor graphics capabilities. Most are able to render a single 2D terminal with a decent amount of latency, and that's about it. As a result, we are using a variety of Linux-based benchmarks to test the Xeon E5-2600 v3.

If you've manually configured and run benchmarks under Linux, then you know they can be an exercise in frustration. For this review, we are using a "simple" test script to automate running a few common Linux benchmarks. It's free and can be found at linux-bench.com or on GitHub. There is also a new Docker.io version of the script on GitHub, so it can be run using perhaps the hottest technology of the year. The bonus of using this type of script is that it is freely available to test on your own server.

It is designed to run off of a standard Ubuntu 14.04 LTS LiveCD using only three commands. As a result, it can even be run remotely on servers with no local disks installed via KVM-over-IP. We did run with a local LiveCD image, booting into a CLI environment before each iteration to ensure no artifacts were leftover from previous runs.

The Linux-Bench script itself does little other than install dependencies and run benchmarks. As a disclaimer, I am one of the community contributors to the script. However, I do not help maintain any of the individual benchmarks.

UnixBench 5.1.3

The byte-unixbench project can be found on Google Code here. However, it has roots dating back to 1983. It is an extremely popular suite that has a number of component tests like Dhrystone, Whetstone, and shell scripts. Specifically, we are interested in the CPU tests, so metrics for 2D/3D GPU and storage are excluded. Also, since these systems have many processing cores, we utilized the high CPU count patch.

c-ray 1.1

c-ray 1.1 is a popular and simple ray-tracing benchmark for Linux systems written by John Tsiombikas. It is designed so that, on most systems, it should not need to access RAM and therefore is highly sensitive to processor performance. You can find archived results, including those from SGI systems, here.

STREAM

STREAM is perhaps the seminal memory bandwidth application. The benchmark was created and is maintained by Dr. John D. McCalpin. More information can be found here.

OpenSSL

OpenSSL caused a stir with the now famous "Heartbleed" bug earlier in 2014. This is the technology that secures much of the Internet's data traffic, and is a common server application.

HardInfo

HardInfo is a simple benchmark that's popular in Linux-based environments. It is well known perhaps because the benchmark is installed by default on many Ubuntu desktop systems.

NAMD

NAMD is a molecular modeling benchmark. It was developed by the Theoretical and Computational Biophysics Group in the Beckman Institute for Advanced Science and Technology at the University of Illinois at Urbana-Champaign. More information can be found here.

NPB

NPB or NAS Parallel Benchmarks are a set of computational fluid dynamics applications originally intended to benchmark parallel supercomputers for NASA. We are using only one node for our testing, though today's multiprocessor systems in some ways mirror parallel computers from many years ago. You'll find more information on NASA's site, here.

p7zip

7-Zip is a popular open source compression application. Servers compress data for storage purposes and also before transmitting. It is an extremely common tool and common application.

Redis

redis is a popular new Web technology to help online applications scale. This is an in-memory key value store, making it memory bandwidth- and CPU performance-bound. It's an emerging technology with a strong developer base.

Sysbench CPU

Sysbench is another venerable benchmarking application. It is extremely easy to use and, for this test, we are only focusing on CPU performance.

You can easily replicate these tests by downloading and running them individually. Using the Linux-Bench script's parameters have been profiled across over 100 different systems, from low-end Atoms up to quad-socket Xeon and Opteron servers, thanks to an active community participating and posting edits in GitHub.

Benchmark Results

For our first set of benchmarks, we are going to look at the most common suites we ran, including UnixBench (both in single and multi-threaded modes), HardInfo, sysbench, and STREAM.

UnixBench 5.1.3

One way that Intel keeps thermals manageable on the more complex Haswell-EP-based CPUs is scaling back clock rate. For example, the Xeon E5-2699 v3 operates at just 2.3 GHz, which is 300 MHz less than the -2690 v3. Single-threaded performance is still highly relevant in server workloads though, which is why Turbo Boost technology exists. A great example of this is Minecraft, which went from an obscure title to a phenomenon. The game server was bottlenecked by single-threaded performance, compelling many admins to use Xeon E3s in a quest for higher frequencies.

In our first UnixBench Whetstone/Dhrystone run, we ran the test in single-threaded mode.

Single-threaded Whetstone is relatively consistent between the three processors, despite a 700 MHz difference between the base clock rates of Intel's Xeon E5-2690 v2 and -2699 v3.

Single-threaded Dhrystone is a different story; the Xeon E5-2690 v1 pulls ahead by almost 10%. Despite the scaling of this chart, however, the results are really fairly close, even if we'd typically expect the architectural improvements rolled into Haswell to convey significant advantages over Sandy Bridge.

We can turn to the multi-threaded results to see more notable changes.

As we might expect, the threaded results illustrate that adding cores helps scale performance in workloads properly optimized for multi-core designs. The Xeon E5-2699 v3 puts up greater than 2x performance improvement versus the -2690 v1, which was top-of-the-line in its day.

We clearly see the evolution of Intel's Xeon E5-2690 line-up from its first iteration to the v3 version. The other standout is the Xeon E5-2699 v3, which shows that 18 cores and 36 threads per processor deliver huge gains in a parallelized task, particularly compared to the once-fastest Xeon E5-2690.

HardInfo

This is certainly less dramatic than our Whetstone and Dhrystone results, but there is still solid scaling.

Our next tests are the Fibonacci sequence calculation and FPU FFT module.

Higher core counts again benefit the v3 processors.

In all three metrics, we see linear improvements from one generation to the next, as the Xeon E5-2699 v3 pulls ahead. Intel's original Xeon E5-2690 was a 2.9 GHz part, and the -2690 v2 stepped up to 3 GHz, so the fact that lower-frequency v3s maintain a lead is telling.

Sysbench CPU

Searching for prime numbers is a math problem that can be parallelized easily. As a result, it scales well with additional cores.

The Haswell-EP parts are on par with Sandy Bridge-EP and Ivy Bridge-EP when it comes to single-threaded performance. Of course, we know from the growing core counts that Intel is putting its emphasis on extra execution resources, rather than burning TDP on peak clock rates. So, maintaining the status quo there was likely deemed acceptable. But load down all available cores and it's easy to see where Haswell-EP has its greatest impact.

STREAM

We did make one adjustment to the test configurations before running these tests. After noticing our control server with the Xeon E5-2699 and Supermicro-sourced boxes were scoring similarly, we decided to create a little side experiment, giving the -2699 v3 four 16 GB DDR4 DIMMS per processor. The Xeon E5-2690 v3 received eight 8 GB RDIMMs per processor to match the first-gen and v2 platforms.

The results show both the impacts of adding more memory and the nice scaling we get moving from 1600 MT/s DDR3L to 2133 MT/s DDR4. There is clearly a performance benefit attributable to the new standard; it's not just about power consumption.

More Benchmark Results

Next up are our application-specific benchmarks, including c-ray 1.1, NAMD, NPB, p7zip, redis, and OpenSSL. At some point in the future, optimizations for Haswell-EP's advanced instructions may find their way into these titles. But for now, the performance we're reporting represents the current state of affairs. Specifically, AVX 2.0 would likely have a major impact on the results.

c-ray 1.1

Linux-Bench actually runs three different c-ray tests. The first is dubbed "easy", and is great for showing performance differences between Atom processors and desktop CPUs. We excluded that measurement because all three platforms finish it in under one second. Instead, we are using the much tougher command cat sphfract | ./c-ray-mt -t $threads -s $resolution -r 8 to demonstrate differences between these platforms.

Ray tracing generally scales well with both CPU frequency and core count; we see both trends in action as the Xeon E5-2600 v3s pull ahead.

While the 1920x1200 test responds readily to more execution resources, the 3840x2160 benchmark doesn't. Some of that may be due to the -2690 v3's 300 MHz per core advantage. Still, the scaling of the Xeon E5-2690 from one generation to the next is made obvious.

NAMD

Our NAMD tests use molecular modeling to tax these server platforms. For anyone involved in projects like Folding@Home, these are the types of workloads that fully utilize multi-threaded processors.

Haswell-EP has little trouble showing off its strengths.

The first-gen Xeon E5 and v2 results aren't what most folks would expect. However, Ivy Bridge-EP had a nasty habit of getting aggressive on power-saving, dropping all cores to lower P-states when demand dropped. That may be what's happening here. In contrast, the Xeon E5-2600 v3s control this on a per-core basis, so the impact of turning cores on and off isn't reflected as painfully in the performance benchmarks.

NPB

For a test with "Parallel Benchmark" in its name, we're expecting Haswell-EP's high core counts to yield big performance numbers. 

hereas we see relatively pedestrian improvements going from first-gen Xeon E5-2690 the Haswell-EP-based variant, Intel's -2699 v3 finishes way ahead of the other CPUs. Since this was repeatable, I'm hypothesizing that the problem being solved fits into the big die's 45 MB L3 cache.

P7zip

p7zip is a standard compression benchmark. Generally, these types of algorithms are able to take advantage of many threads. I'd guess that the Haswell-EP parts are able to overcome small frequency deficits to finish with a lead, thanks to their IPC throughput advantage and core count.

There is a linear-looking performance improvement stepping between each generation of Intel's Xeon E5-2690. The Xeon E5-2699 v3 again shows off what extra cores can do in a workload able to utilize them, posting an approximately 2x increase over the first-gen Xeon E5-2690.

Redis

Redis is an in-memory application, so core count has less of an overall impact.

As I expected, the results fall much closer to each other, looking a lot like our STREAM results. Still, the configuration with one 16 GB DDR4 DIMM per channel does pull ahead.

OpenSSL

Again, OpenSSL is widely used, so this is perhaps one of the most applicable benchmarks for Web servers. Some companies are pushing for broader use of SSL to keep data encrypted, making the metric particularly important.

The Haswell-EP-based parts scale well. In particular, the Xeon E5-2699 v3 shows a greater than 2x performance improvement over Intel's once-top-of-the-line Xeon E5-2690.

Power Consumption Results

Intel has this cadence where its latest architectures roll out on the mobile/desktop side, and are then followed in the high-end workstation/server space. We saw the benefits of 22 nm manufacturing first from Ivy Bridge, and then with Ivy Bridge-EP. Then, Haswell integrated the platform's voltage regulation circuitry for tighter control of power on-package. Needless to say, we were expecting a notable improvement from Haswell-EP to follow some of the gains already measured using desktop-class offerings, and we got it.

Haswell-EP incorporates a number of technologies that we anticipated bettering the power consumption story. First, it brings much of the power delivery from Intel's first- and second-gen Xeon E5s on-package. Haswell-EP can control P-states on a per-core basis, allowing them to be granularly spun up and down as demand dictates. Intel claims up to 36%-lower power consumption from its Per-Core P-States (PCPS.)

Then there's DDR4 memory, which, in addition to increasing data rates, also employs a lower input voltage. In most desktops, sub-1 W savings per module doesn't mean much. In a server, however, where you might have eight DIMMs per CPU and multiple processors per node, hacking away at total platform power a few watts at a time adds up quickly.

For companies looking to upgrade from three-generation-old Westmere-EP processors, there is another major difference. The new Wellsburg PCH (Intel C610 series) runs extremely cool and can control 10 SATA-based drives using standard ports. Back when Westmere was modern, the only way to get lots of PCIe connectivity in a server for expanded storage was adding a second IOH36 chip. That component required a decent sink and plenty of airflow to cope with its heat. When two of them were in a system, they became a significant cooling consideration. Since Haswell-EP employs 40 lanes of PCIe 3.0 on-die, and is mated to a C610 PCH manufactured on a newer process, you get big platform power savings compared to pre-Sandy Bridge-EP systems. Servers sporting Westemere-EP started coming off of lease en masse in early 2014, so they're the ones most likely to be replaced by Haswell-EP.

To generate some hard data, we took our 1U Supermicro test bed and allowed it to idle with no PCIe expansion cards installed (only the on-board networking controllers were active). The results were awesome:

Remember, our Haswell-EP-based server sports two 135 W Xeon E5-2690 v3s and 16 eight-gigabyte memory modules. It also uses redundant cooling, which is great in a datacenter environment, but not particularly power-friendly. Even still, the takeaway is that the Haswell-EP-based system's idle power consumption is extremely low.

Then we fired up three instances of well-threaded tests using c-ray 1.1, sysbench CPU (prime solver), and STREAM concurrently. The results were interesting; mainly, the Xeon E5-2699 v3 drew quite a bit more power than we were expecting. Granted, in many of our performance benchmarks, those same CPUs deliver greater than 2x improvements over the first-gen Xeon E5-2690. That's what will trigger consolidation of older machines into fewer Xeon E5-2600 v3-based boxes during this refresh cycle.

Haswell-EP Evolves The Server And Workstation

The night Chris Angelini was writing his review of the Core i7-4770K for Tom's Hardware and I was doing the same, we both reached similar conclusions: Haswell on the desktop is not a big deal. But Haswell-EP is a completely different story. Intel uses its advanced manufacturing to enable more cores, more cache, and a redesigned memory controller able to support DDR4. All of that comes together to yield a big step up compared to Ivy Bridge-EP. When you consider that these CPUs replace parts in servers with four to eight cores, the potential gains are substantial. Delivering twice the performance in a similar form factor makes it easy for any business to at least consider consolidating their hardware infrastructure.

When it comes to power consumption, we already know that Haswell was designed to service the mobile space. This has some favorable implications in the server world too. Of course, the difference is that Haswell-EP-based CPUs are much larger (and multiplied in a dual-socket configuration), so all gains are amplified.

In terms of performance per core, unless your software is optimized to exploit AVX 2.0, Haswell's biggest benefits come from the architecture's inherent IPC tweaks. Where Haswell-EP really shines is its higher core counts that help scale performance accordingly in well-parallelized workloads.

DDR4 memory support is perhaps the most next-generation aspect of Intel's new Xeon E5-2600 v3 processors. In time, we will likely see higher data rates, increased density, and potentially lower-power versions of the standard. Unlike DDR3, DDR4 is still supply-constrained, so new servers are going to be priced higher until memory vendors catch up. Right now, the market is split. Most consumer devices are tied to DDR3; Haswell-E/EP is the first design pushing DDR4. That'll change slowly. But for now, there are quantifiable power and performance benefits to justify the eventual adoption of a what currently appears to be a ridiculously expensive technology.

Reflecting on the press day that Intel hosted to introduce Haswell-EP, higher core counts, DDR4, and advanced ISA support were the most obvious platform changes. But the company's Fortville adapters are arguably even more exciting to me. The doors opened by a low-power controller capable of two 40 GbE interfaces or eight 10 Gb links cannot be ignored. I have been using Mellanox ConnectX-3 VPI adapters for quite some time in 40 Gb Ethernet mode. But the power consumption benefits of Intel's technology compelled me to go out and buy a new 40 Gb Ethernet switch.

Truly, this is the march of progress. More IPC throughput, a greater number of cores, more memory, and beefier I/O to exploit the platform's bolstered data handling capabilities translate to further consolidation of workloads. Intel is clearly driving towards a software-defined vision and takes a major step toward that goal with its Xeon E5-2600 v3 introduction. Then again, the way Intel presents its strategy addresses a more complete datacenter solution. Much like HP, Intel no longer pitches the Xeon as a new, faster processor on its own (even if it is). Instead, the company has a holistic goal for driving compute, storage, and networking performance over the next few years. Haswell-EP is the showcase for that.

Maximizing The LGA 1150 Mainstream?

As enthusiasts, when we pay more, we want to get more. That's not too much to ask, right? That principle certainly applies to the high-end motherboards in today’s round-up. For example, we find a second Ethernet controller on four of the five solutions. They all sport add-on storage controllers. And all but one also include expanded USB 3.0 capabilities beyond the Z97’s paltry six ports.

Three of today’s contenders even spread PCIe 3.0 connectivity across three slots for improved triple-card CrossFire performance (even if none of the board's in our story are quite pricey enough to include the extra bridge logic needed to make them three-way SLI-compatible).

Well, that’s something at least.

We also expect the best possible overclocking performance in this price range, so we’ll test that out using Intel’s new “Devil’s Canyon” Core i7-4790K. Which board can take home the prize?

ASRock Z97 Extreme6

ASRock exposes a bunch of traditional high-end features to the I/O panel of its Z97 Extreme6, yet doesn’t skimp on the modern stuff either. Getting rid of the VGA connector, for example, gives the board room for eSATA and a CLR_CMOS button. If you really need VGA output from the integrated GPU, it's accessible through a DVI-I adapter block (not included).

Packing the I/O panel with six USB 3.0 ports doesn't leave much space for legacy USB 2.0, though two of the newer ports are connected directly to the chipset’s controller for the best possible legacy compatibility. We also find a pair of gigabit Ethernet ports, and we certainly wouldn’t give away any of those in exchange for slower USB.

The Z97 Extreme6 has a nice bundle of internal features as well, beginning with its ten SATA 6Gb/s ports, two M.2 slots, SATA Express, and mini-PCIe. Yet, there is a little bit of visual cunning here, as the super-fast PCIe x4-based Ultra M.2 doesn’t interface with any of the chipset's SATA connections. Moreover, the mini-PCIe slot merely resembles mSATA, and the one extra connector that does have SATA ports—the second M.2 connector—uses the same ports as SATA Express. Even eSATA on the I/O panel functions only to the exclusion of one of the internal ports.

A total of ten SATA ports is still fairly generous for an LGA 1150-based platform though, and ASRock gets to that number by adding two PCIe-based controllers. Other PCIe-attached subsystems, such as the secondary USB 3.0 and gigabit Ethernet controllers, along with PCIe x1 slots and one of the two M.2 slots, all share the limited wealth of the Z97’s eight PCIe 2.0 lanes. ASMedia’s ASM1184e PCIe 2.0 four-lane switch keeps most of the Z97 Extreme6’s pathways active, excluding just one of the PCIe x1 slots when the mini-PCIe slot is used.

The primary “Ultra M.2” slot might not have SATA, but its four PCIe lanes are all third-gen-capable. That means they come from the CPU’s 16-lane controller. And that, in turn, means that builders must choose between having one graphics card fed by sixteen lanes, two cards fed by eight lanes, or two graphics cards and an M.2 SSD in a x8/x4/x4 configuration. Should you install a graphics card and drop a drive in the Ultra M.2 slot, you'll simply waste the four lanes that feed the second graphics slot, unless another add-in is used there.

All of those factors help explain why this isn’t a three-way SLI-capable board. In fact, we're hesitant to call it a platform you'd drop three cards into at all. The Z97 Extreme6’s third x16-length slot is fed by only two PCIe lanes, and both are stuck sharing bandwidth on the PCH’s bandwidth-constrained second-gen controller.

A pair of internal USB 3.0 front-panel headers serves up to four front-panel ports, which appear perfect for certain high-end cases like the custom-configured Merlin SM08 we tested. You might say that the Z97 Extreme6’s second header is useless whenever it’s concealed by a third graphics card, but let’s be frank: again, that bottom slot isn’t suitable for the type of high-performance card you'd want to put there.

The Z97 Extreme6 likewise has dual firmware ROMs with a selector switch that lets you flip from one to the other if you completely mess the first up. And ASRock even socket-mounts both ROMs, just in case you need to perform a replacement.

Four SATA cables seem a little scant on a board that has ten ports, but it’s still enough to finish most of our high-end builds.

Z97 Extreme6 Software

ASRock’s A-Tuning software is unchanged for the Z97 Extreme6, though our use of a different processor does change a few of its configuration menus. “Advanced Turbo” configurations for our Core i7-4790K include a 4.5 GHz clock at 1.21 V core, and a 4.60 GHz clock at 1.26 V core. Both crashed almost instantly at full load when we tried to use them.

“Optimized CPU OC Settings” likewise included 4.5 to 4.7 GHz overclock at 1.2 to 1.34 volts. All crashed under eight threads of AVX-optimized Prime95, but at least the 4.7 GHz at 1.34 V configuration held up for 20 minutes.

The OC Tweaker menu still provides the full range of settings found within motherboard firmware, and we confirmed that voltage/frequency/CPU multiplier settings functioned normally.

A-Tuning also has a nicely detailed system monitoring page that beats most of its competitors in the number of readings. Some items, such as DRAM voltage, weren’t as accurate however.

Z97 Extreme6 Firmware

I mentioned having a new CPU to use for this round-up, and that processor is Intel’s Core i7-4790K. Intel made a lot of noise in March’s pre-launch briefing, but the clamor was muffled by its Computex launch. It turns out that the improved thermal material was no magic bullet; the die in our sample has worse voltage handling characteristics than the Core i7-4770K I was using previously.

Our general recommendation to stay under 1.3 V over the long term is based on reports from readers, motherboard manufacturers, and professional overclockers, and that remains. In the past, however, we were thermally limited by our CPU at 1.25 V. That constraint shifts to 1.28 V with the -4790K we're using today. Unfortunately, this sample requires that extra voltage to reach the same 4.6 GHz clock rate. For shame Intel, for shame.

Level 1 CPU load-line calibration keeps our core voltage stable, though the Z97 Extreme6’s 12-phase voltage regulator is certainly part of that stability equation. ASRock is up to its old DRAM overclocking tricks by under-reporting voltage, though we did reach DDR3-2883 at the corrected, 1.625 V setting.

Z97 Extreme6 firmware includes a menu for ASRock’s Thunderbolt add-in card, plus a range of “Tool” menu items, such as its spin-up utility for rarely-accessed hard drives.

Another firmware item facilitates direct UEFI access to ASRock’s email-based tech support.

“Easy Driver Installer” lets you grab the newest drivers directly from ASRock’s server prior to loading the operating system.

Asus Z97 Pro(Wi-Fi ac)

The (Wi-Fi ac) suffix adds Broadcom’s dual-band 802.11ac controller to the already-likeable features of Asus' Z97 Pro. That controller also features integrated Bluetooth connectivity, and stands in contrast to the secondary wired controller offered by competing products in today’s round-up. We'd certainly call this board's networking suite most-practical.

The wired controller, Intel’s WGI218V, leverages PCH features over a custom link to save a PCIe lane. That means the only “waste” we see on the I/O panel is a VGA port, which we saw ASRock smartly do away with.

Other I/O panel features include four USB 3.0 and two USB 2.0 ports, HDMI, and DisplayPort. A bit legacy for most enthusiasts, those “slow” USB 2.0 ports are still fast enough for a keyboard and mouse. And, users of ancient keyboards or mice will also find a PS/2 port that can serve either purpose.

Asus TPU (auto overclocking), EPU (lower-energy mode), and EZ XMP switches stand in stark contrast to a black PCB. That last switch lets you invoke an Intel XMP memory profile without forcing you into the firmware; it could be useful for boutique builders who worry about enthusiastic customers getting lost after resetting the UEFI.

Three buttons make this platform easier for overclockers to use. The Z97 Pro(Wi-Fi ac)'s power button lets you test the board outside of a case without shorting the power pins manually. MemOK allows users of poorly-programmed memory to boot once at lower clock rates and/or higher latencies. At that point, you can enter the UEFI and manually dial-in voltage. And USB BIOS Flashback employs its own logic controller to re-flash firmware, even if you don’t have a CPU or memory installed.

Barely cheaper than more purpose-built three-way SLI boards, the Z97 Pro(Wi-Fi ac) can deliver 16 PCIe 3.0 lanes to the primary slot or split them between the first and second x16 slots in x8/x8 mode. The third x16-length slot gets a maximum of four lanes from the Z97’s eight-lane PCIe 2.0 controller, though two of those lanes are also wired to x1 slots and one more is wired to the ASM1042AE USB 3.0 controller. An ASM1184e PCIe x1 to x4 switch gives the Z97 Pro(Wi-Fi ac) enough connectivity to feed other on-board devices, but still doesn’t have enough lanes to fix that specific pathway sharing issue. If you want to use any of those aforementioned features, the bottom slot shifts into x1 mode. Visual tricks like this might attract customers, but I would have settled for an open-ended x1 (or x4) slot in that location instead.

Two of the Z97 PCH’s six SATA ports are found within a SATA Express connector, so they can be used either way. Those same ports are, however, also connected to the M.2 interface, forcing some builders to choose between connector technologies. Asus retains six traditional SATA-only ports by adding a two-port ASM1061 SATA 6Gb/s controller (though both ports share a single 5 Gb/s PCIe 2.0 link).

The Z97 Pro(Wi-Fi ac) front-panel audio jack is found about an inch forward from the traditional rear corner location. The significance of this layout change depends on case selection, since some cases still have FP-Audio cables around half-an-inch too short to reach that corner. It’s hard to believe that certain case manufacturers still haven’t addressed the issue (a carryover from AC'97 days), but Asus has.

Four SATA cables in the Z97 Pro(Wi-Fi ac) installation kit appear a little basic, especially given the unit's $212 price. Then again, Asus appears to be targeting a gaming and overclocking market that often isn’t very reliant on complex drive configurations.

Z97 Pro(Wi-Fi ac) Software

Asus Ai Suite hasn’t changed noticeably from our previous discussion, though the combination of wireless controller and a new test processor give us a few things to discuss in this Z97 Pro(Wi-Fi ac) review.

The firm’s five-way optimization protocols brought our processor to 4.6 GHz at 1.33 to 1.35 V, which slightly exceeds both our recommended 1.30 V limit for continuous operation of fully-loaded Haswell-based cores, and our CPU cooler’s capability. Your processor and use scenario may make these settings more agreeable.

Ai Suite still offers fan speed and voltage regulator controls to save noise and/or energy as the user desires.

Software controls are available within firmware limits for base clock, CPU ratios, and most of the Z97 Pro(Wi-Fi ac)’s voltage levels. Better still, all of the adjustments we made (within our CPU’s capabilities) worked.

EZ Update links AI Suite directly to Asus’ driver servers. EZ Update’s “USB BIOS Flashback” tab to puts the latest Z97 Pro(Wi-Fi ac) firmware onto your USB thumb drive.

Asus Wi-Fi Engine let you turn your 802.11n Wi-Fi transceiver into an AP, using gigabit Ethernet to reach the Web.

Asus continues to provide a quick-charging USB port option for BC1.1-compliant portable devices, turbo mode for USB 2.0, UASP performance enhancements for USB 3.0, and traditional quick-charge mode for non-BC1.1 devices.

A Web extension for Ai Suite system monitoring, Asus Push Notice is accessible through downloadable iPhone and Android apps.

More than just a basic status report, Asus System Information also provides the most comprehensive memory programming information we’ve seen. Ai Suite’s software version tab is similarly complete.

Z97 Pro(Wi-Fi ac) Firmware

Though we find XMP and EZ O/C settings on the firmware's starting page, we’d rather jump into its finer features at the touch of an F7 key.

Though the screen shot shows a 46x multiplier, we were truthfully able to reach 4.59 GHz at 45x 102 MHz. Unfortunately, the DRAM only allowed us to reach a 101 MHz BCLK at 28x, so the displayed settings are closer to the actual results.

We measured a 1.28 V CPU core at the motherboard’s actual 1.28 V setting. That kind of accuracy is appreciated. On the contrary, our DRAM reached 1.65 V at the Z97 Pro(Wi-Fi ac)’s 1.625 V setting, and firmware showed only 1.646 V.

Primary, secondary, and tertiary timings are all adjustable. The Z97 Pro(Wi-Fi ac) even used our DRAM’s XMP settings as the baseline (when set to XMP mode).

Also known as droop compensation, the Z97 Pro(Wi-Fi ac) features CPU Load-Line Calibration to reduce voltage sag under full load. Default settings worked for us, though we did increase the power menu’s CPU current threshold by 20%.

Asus Easy Tuning Wizard appears to be a firmware-based overclocking utility designed for people who don’t want to overclock. As confusing as that sounds, we were more confused when the system wouldn’t boot after using it. That’s because the company's software-based automatic-overclocking worked well.

Gigabyte Z97X-UD5H

Gigabyte packs its Z97X-UD5H with the dual networking controllers and six USB 3.0 ports of ASRock’s competing sample, plus the two USB 2.0 ports and VGA output of Asus’ board. How?

Though it probably has most of the features I'd personally use, the Z97X-UD5H is missing ASRock’s eSATA and I/O panel-based CLR_CMOS buttons, as well as the DiplayPort output both competitors expose. Gigabyte still manages to fill more space compared to Asus, though that company's second network connector is wireless. Keeping the balance of features in mind, Gigabyte also prices its Z97X-UD5H between the aforementioned competition.

The Z97X-UD5H is a completely different kind of product internally though, primarily because of its triple-graphics-optimized PCIe x16 slot configuration. Enthusiasts can switch between one card with 16-lane transfers, two with eight-lane links, or three cards in a x8-x4-x4 mode, all driven by the CPU's 16-lane controller at third-gen signaling rates. Though Nvidia precludes this configuration from three-way SLI compatibility, it works well in three-way CrossFire. AMD on Intel for the win?

Optimizing slots for CrossFire does have one disadvantage, though: any card placed into the third x16 interface kicks the middle slot into x4 mode. That’s something to keep in mind if you want to put something other than a graphics card there, especially since Nvidia dictates that you can't use x4 slots for SLI. Anyone with two GeForce cards will want to pretend that the Z97X-UD5H’s bottom slot isn’t even there. This is the same problem anyone buying ASRock's board faces if they decide to use the exclusive “Ultra M.2” feature, which borrows four PCIe 3.0 lanes. So, the debate between more graphics cards or faster storage begins here.

Keeping CrossFire in mind for the third slot, Gigabyte wisely omits bottom-edge USB 3.0 headers. We instead find a single two-port header near the middle of the front edge—right behind the diagnostics code display—where accessibility remains uncompromised.

SATA Express cables use two of the Z97X-UD5H’s eight SATA 6Gb/s ports. As with the previous competitors in today’s round-up, those two ports are lost whenever an M.2-based drive is installed. Unlike the other two companies, though, Gigabyte saves a PCIe 2.0 pathway by putting two legacy PCI slots on a single-lane PCIe-to-PCI bridge.

A closer look at the controllers shows that Gigabyte saves one more PCIe lane by putting the second gigabit Ethernet port on Intel’s proprietary-interface network controller. Getting some of its resources from the Z97 PCH, this WGI217V PHY would normally be called a primary controller, except that Gigabyte reserves the honor for Qualcomm’s PCIe-based Killer E2201. Enthusiasts who utilize the firm’s packet prioritization technology will hail the move, while those who don’t are left wondering why Gigabyte chose a combination of dissimilar parts that doesn’t support teaming.

The corner in front of the DIMM slots is crowded with a bunch of tuner-friendly features, including a row of component voltage detection points, a power button for use without a case, a reset button for the freeze-ups caused by overzealous overclocking, a CLR_CMOS button to rid firmware of settings that won’t let it POST, a firmware ROM selector switch the helps power users get around corruptions, and a dual-BIOS mode selector that, from my experience, doesn’t. I leave that last capability turned off.

Gigabyte serves up the third motherboard in this round-up that includes only four SATA cables in spite of its large port count. That cost-saving move pushes value towards gaming enthusiasts by taking it away from storage enthusiasts, but perhaps the majority of storage enthusiasts have plenty of spare cables already?

Z97X-UD5H Software

Gigabyte EasyTune hasn’t changed much in the past year or two, but the company does help us include more readable screenshots by reducing the size of its GUI. Default overclocking profiles for the tested Core i7-4970K include 4.60 GHz at 1.35 V CPU core, 4.80 GHz at 1.50 V, and a 5.00 GHz setting that we didn’t even try for fear of frying our CPU.

An automatic tuning algorithm nudged our processor up to 5.10 GHz. Like the other “Smart Quick Boost” settings, the software wasn't as smart as it purported, and the only quick thing about it was the speed with which our system crashed upon the application of a heavy software load. Each CPU is unique though, so perhaps you’ll have better luck.

EasyTune’s advanced settings mostly work, we were able to measure voltage changes at the Z97X-UD5H’s voltage detection points, and CPU-Z was able to detect set changes in base clock and CPU multiplier. DRAM multiplier changes still require a reboot, and DRAM timings are still locked out.

One of the ways Gigabyte shrank its EasyTune menus was placing its hardware monitor on the right edge of the screen, at full height. I cropped this image before shrinking it.

Unlike most system information pages, the one in Gigabyte EasyTune adds fan controls. You can choose between automatic and manual methods, including setting your own temperature-to-RPM slope.

Z97X-UD5H Firmware

Gigabyte still prefers to spread its firmware-based overclocking settings over far more pages than we believe are necessary (or even convenient), with an M.I.T. menu that opens to nothing more than a list of submenus and a simple status report.

Though the Z97X-UD5H needed more than our chosen 1.28 V to reach 4.6 GHz, it reached 4.54 GHz at that target voltage by choosing a 45x multiplier and 101 MHz BCLK.

Changing “Memory Timing Mode” to “manual” allows tweakers to set the primary, secondary, and tertiary timings of both channels simultaneously. “Advanced Manual” mode supposedly allows users to set per-channel timings, but we fail to see the point of doing this on a dual-channel motherboard. Fortunately, current values are shown next to manual settings so you won’t forget the baseline.

Voltage controls that could easily have fit in a single page are instead broken up across four sub-submenus found within the “Advanced Voltage Settings” submenu. We reached 1.28 V CPU core at the 1.25 V setting, and 1.65 V at the 1.63 V DIMM setting.

MSI Z97 MPower

MSI cleans up the I/O panel of its Z97 MPower by removing both DVI and VGA, instead focusing on outputs like HDMI and DisplayPort without sacrificing its six USB 3.0 ports and four USB 2.0 ports. Even the analog audio section features six stereo jacks, and the clear-settings button is clearly visible. Yet, there’s still a hole in the port arrangement. What’s missing?

The Z97 MPower is the only board in today’s comparison featuring a single networking controller. Even Asus covers secondary networking via a wireless controller, and I have to imagine that most enthusiasts will find the combination of GbE and 802.11ac superior to the implementations of MSI's other competitors. Even though a second gigabit Ethernet port wouldn’t have been a high-priced add-in, we were still a little surprised that the Z97 MPower costs slightly more than most competing products. So, we began looking for other value-boosting features.

There is, for example a SATA Express adapter card in the M.2 slot, a value add that MSI also sells separately. MSI even said a month ago that it would begin shipping the Z97 MPower with that adapter bundled. However, Newegg still shows the original adapter-free package.

The Z97 MPower also offers a row of voltage detection points, eight SATA ports, and several buttons and switches to ease overclocking from an open bench. While most of those features are also offered by Gigabyte, MSI’s O/C Genie and base clock control buttons stand apart. Some tuners might even like that MSI’s voltage detection points have pins rather than solder dots, and that the installation kit includes a set of cables for connecting those pins to an external voltmeter.

Comparisons to Gigabyte’s sample don’t end there. MSI is the second company in this round-up to offer x8/x4/x4 pathway configurations to host three PCIe 3.0 graphics cards in CrossFire. Nvidia’s resistance to enabling SLI on four-lane links isn’t brand-specific, so it affects both competitors equally. The same caveat also applies: enthusiasts who want the second PCIe 3.0 slot to retain eight lanes (and SLI capability) must leave the bottom slot empty.

MSI adds a fourth x16-length slot at the center of the Z97 MPower, which shares three of its four PCIe 2.0 lanes with the three x1 slots. Using any x1 slot means treating this one equally, as a x1 slot. And if you actually need to install an extra graphics card along with some x1 cards, at any bandwidth cost, MSI uses an open-ended connector in the bottom x1 slot.

Like most competitors, M.2 and SATA Express early adopters are forced to give up two of the chipset’s SATA 6Gb/s ports in order to keep their preferred interface active. Likewise, MSI relies on a PCIe 2.0 single-lane controller to serve up the board’s seventh and eights SATA 6Gb/s ports at a combined 5 Gb/s.

MSI tackles the problem of stiff USB 3.0 front-panel cable ends getting in the way by using a right-angle header, conveniently placed above the top-most SATA ports, where the cable can snake behind the lower drive cage of most enthusiast-class ATX cases. It also moves the front-panel audio header around ½” forward from the bottom-rear corner, as it happens that many cases have a cable that’s ½” too-short to reach there. Many builders will still need to perform stretching exercises to get that cable over its header, however.

The Z97 MPower costs only $5 more than its closest competitor, yet has a similar feature set. Fortunately, it also includes two extra SATA cables for a total of six. An extra $2.50 per cable isn’t bad.

Z97 MPower Software

MSI Command Center shows minor updates nearly every time we install it, and usually needs to be updated after a board launches in order to support its newest features. Fortunately, Z97 is mature enough that Command Center updates have caught up.

CPU ratio and BCLK adjustments work, but altering them can take a while (these values cannot be keyed in). CPU fan controls, including both fixed mode and custom slopes, are also found on Command Center’s opening page.

CPU voltage adjustments are available on the second page of MSI Command Center’s opening menu. Adjusted values can be confirmed through both software and hardware, since the Z97 MPower has voltage detection points. DRAM settings in the adjacent box can’t even be selected, let alone changed.

The Command Center suite includes MSI RAMDisk, which we also confirmed working.

Command Center’s “Advanced” tab provides access to a voltage menu that’s roughly equal to the settings available from the Z97 MPower’s firmware. We were able to confirm the functionality of several settings via voltage detection points, including input, core, and DRAM voltage.

Another “Advanced” submenu provides system fan controls in both temperature-based and fixed modes.

An advanced DRAM timings menu makes firmware adjustments from software, requiring a reboot.

MSI Command Center also has a logging utility for several of its monitoring functions.

Mobile Control allows users to access several Command Center features, including monitoring and some overclocking controls, using a smartphone. I was able to get the associated app running on my ZTE Android phone and take control of my PC in minutes.

Z97 MPower Firmware

MSI continues using oddly-scaled fonts and buttons in its firmware. But at least this configuration has become familiar to most of its enthusiast customers. MSI also spreads its settings across far fewer menus, for far quicker access, compared to Gigabyte.

A 1.28 V CPU core was measured at the Z97 MPower’s 1.24 V setting, and was reported by the BIOS as 1.256 detected volts. DRAM similarly required a 1.62 V setting to reach 1.65 V. Those voltage levels allowed our CPU to reach 4.5 GHz, but no more.

DRAM reached approximately 2828 MT/s by adding 1 MHz base clock to the DDR3-2800 XMP profile.

The Z97 MPower’s CPU Features menu helps overclockers disable any capabilities that get in the way of a stable overclock.

Setting “Link” mode for DRAM allows both channels to be configured simultaneously, rather than separately. Primary, secondary, and tertiary timings are all configurable, with previous values shown in grey next to the new setting you’ve made.

Supermicro C7Z97-OCE

After a couple of attempts to break into the enthusiast space with Intel's previous chipsets, a more seasoned Supermicro is back with its well-developed C7Z97-OCE. Will the sever/workstation company finally be able to translate its legendary durability into a language overclockers can understand?

The C7Z97-OCE’s I/O panel includes the two network controllers expected on a high-end board, but comes up a little short in USB connectivity (you get four USB 3.0 and two USB 2.0 ports). Perhaps that was due to a shortage of space, caused in part by the VGA connector that could have either been left out altogether or wired through DVI-I.

Intel supplies both GbE controllers, with one using the chipset’s proprietary interface to save a PCIe lane. Supermicro still adds a PEX8605 four-lane PCIe 2.0 bridge, but that’s to maintain PCIe x1 slot availability concurrent to other devices, such as the WGI210AT secondary network controller, an AMS1061 SATA controller, and a two-lane M.2 slot.

The C7Z97-OCE’s slot configuration appears a little extreme by mainstream chipset standards. Those x4 slots are wired as x1, though. And all three x4 connectors are open-ended to support PCIe x8 and x16 cards at reduced PCIe 2.0 x1 bandwidth. That’s in addition to the PCIe 3.0 slots designed for PCIe x16 graphics cards.

Like competing boards from MSI and Gigabyte, the C7Z97-OCE’s three x16-length slots share the CPU’s 16 PCIe 3.0 lanes at 16-0-0, 8-8-0, or 8-4-4 pathways. Unlike MSI, Gigabyte, and the two other rivals in today’s comparison, the C7Z97-OCE does not have a third space between the top two PCIe x16 slots to enable the use of triple-slot graphics coolers. Supermicro instead moves the second and third slot up by one space, making it possible to build a CrossFire configuration using three double-slot cards in a standard (seven-slot) ATX case. I'm a little surprised nobody else thought about that.

Moving the bottom slot up by one space also allows Supermicro to put the front-panel USB 3.0 header along the C7Z97-OCE’s bottom edge, without worrying about it getting in the way of graphics card coolers. That might not be our preferred placement, but at least Supermicro works out an issue that I've criticized on a number of previous products.

Also along the bottom edge is a row of overclocking buttons that support booting with five different profiles labeled for DRAM overclocking; CPU profiles 3, 2, and 1; and “Home”. That last button would ideally be suited to getting your system back up and running after an unstable O/C, but we were still able to get a few settings (such as XMP profiles) saved under the “Home” profile.

Levels 1-3 include factory-programmed overclocks, though all of these appear written for a slower CPU. OC1, for example, dropped our CPU multiplier to a fixed 4.0 GHz, which might have been a nice entry point for the Core i7-4770K (rather than the tested Core i7-4790K).

Getting an overclock started requires a reboot, and you won’t see your tuned settings in the firmware’s GUI until you restart. Fortunately, you can press the button while in that GUI to initiate the reboot and re-enter the GUI to see your settings.

The C7Z97-OCE has six SATA ports. The C7Z97-OCE includes six SATA cables. Bravo. Users need not worry about having left-over cables and useless ports after installing an M.2 drive either, since this is the only board in today’s round-up to have dedicated ports for that connector.

C7Z97-OCE Software

Supermicro relies solely on Intel's XTU (Extreme Tuning Utility) for overclocking within Windows, and includes the standard version on its driver and application disc.

Coming from the server and workstation world, the company's own software looks like it was lifted from a professional management suite. SuperDoctor III client monitors the important input voltage levels and several thermal diodes, but does not monitor the CPU’s internal core voltage.

SuperDoctor III also includes remote management, but I wasn’t able to get past the VeriSign security warning in Windows 8. The “Yes, No, Always” buttons weren’t clickable, and the tab key only switched between the warning pop-up and actual Web page.

C7Z97-OCE Firmware

Supermicro’s basic firmware GUI contains an Overclocking button. But the real excitement starts after engaging Expert mode (to reveal additional settings).

The CPU overclocking menu begins with several overclocking profiles, including 4.6 GHz at 1.30 V. That sounds like a nice limit for long-term use, and it’s even stable. CPU-Z reports a 1.298 V result and, strangely, this relatively high voltage pushed our CPU to only 92° C. That’s a full 6° lower than we expected from the lab's Core i7-4790K at that voltage, given our specific heat sink and fan.

We eventually found a 1.28 V core reading at the C7Z97-OCE’s 1285 mV setting.

Intel XMP mode pushed a pair of our modules to its rated DDR3-2800, but we needed to drop to DDR3-2666 to keep the system stable with four DIMMs installed. Perhaps a little extra voltage might have helped, but our meter insisted that the C7Z97-OCE’s 1.60 V setting yielded 1.65 volts.

Primary and secondary timings are available, and adjusting them from baseline should be easy for anyone with adequate short-term memory to remember what the box said before modification. Dropping from DDR3-2800 to -2666 without re-entering XMP timings was as easy as setting XMP mode first, before switching to manual mode to change the memory multiplier.

Though some documentation showed that the OC3 button should provide customized settings, each field was factory-programmed and re-configurable in firmware. As for the “Home” button, it’s able to keep any settings that were assigned to the base configuration (not assigned to an overclocking profile).

How We Tested Enthusiast-Oriented Z97 Motherboards

A motherboard manufacturer provided our new Core i7-4790K test CPU in hopes that it would really push the overclocking envelope of its motherboard. Unfortunately, that processor's die required more voltage to achieve similar frequencies compared to its -4770K predecessor, killing any chance of us increasing clock rate at safe and sane voltage levels.

Corsair’s Vengeance DDR3-2800 lets us test the DRAM overclocking capabilities of each motherboard with all four slots filled. It unfortunately defaults to DDR3-1333, and default mode lets us disable Turbo Boost cheats that many firms employ when XMP is enabled.

Patriot supplied a different set of 8 GB modules for a former review that defaults to a more performance-friendly DDR3-1600 CAS 9. The 32 GB kit magically transforms into Patriot’s 16 GB dual-channel part number PV316G240C1K by simply removing two of its modules.

PowerColor’s overclocked PCS+ AXR9 290X provides all the performance we need to extract maximum performance from the rest of the platform.

Results: 3DMark and PCMark

Because Intel’s integrated controllers and closely-regulated firmware removes variability between platforms, any sizable difference in the benchmarks should be attributed to either broken settings (like bad memory timings) or unintended overclocking (cheating). Smaller difference can be caused by acceptable differences in base clock (99.8 to 100.2 MHz, for example) or by chance, since differences up to 1% can occur between consecutive runs on the same hardware.

We started to notice a performance advantage favoring Gigabyte’s Z97X-UD5H in 3DMark 11, but the issue really stood out in 3DMark Professional. A quick run through the firmware settings proved there was no way to disable firmware "enhancement" of Intel's Turbo Boost ratios. In this case, the "enhanced" mode forces the CPU's maximum-specified ratio, regardless of the number of cores utilized in the benchmark.

Gigabyte retains a lead through PCMark, though the amount of gain is questionable since this metric is very dependent on storage performance.

Results: SiSoftware Sandra 2014

Gigabyte’s deliberately-forced Turbo Boost ratios are behind those over-the-top scores in Sandra's Arithmetic and Multimedia CPU tests. Remember that this very same overclock is available on most boards, that it is often tied to XMP memory mode, and that we specifically chose not to use XMP mode in order to make this contest fair.

The Z97X-UD5H falls behind in Sandra’s bandwidth-hungry Encoding/Decoding test, and the Memory Bandwidth module shows why: Gigabyte triggers slightly less attractive memory settings, probably in an effort to improve stability with marginal kits.