SberInfra, a technology arm of Sber, Russia's biggest bank, has evaluated the Russian-made MCST Elbrus-8C processors in multiple workloads, but the results were utterly disappointing and the processors failed the test. The testers cited "Insufficient memory, slow memory, few cores, low frequency. Functional requirements not been met at all" as key reasons for the failure. However, there is hope, according to SberInfra engineers.
Homebrew Hardware…
As part of its ongoing conflict with the Western world following the annexation of Crimea in 2014 and the ongoing war against Ukraine, Russia has publicly declared its intention to substitute hardware and software developed in the U.S. and Europe with its domestic technologies. On the hardware side of matters, this meant migrating from x86 AMD's Epyc and Intel's Xeon Scalable platforms to its homegrown CPUs, such as MCST's Elbrus processors based on a proprietary VLIW-like architecture, as well as Baikal Electronics' Arm-based SoCs.
So far, some of the Russian government agencies and government-controlled companies have adopted Baikal and Elbrus-based systems. But when it comes to mission-critical servers, nobody has embraced any of the homebrew machines. This happens to a large degree because most Russian-made machines have fewer cores (compared to mainstream servers), insufficient capacity of slow and outdated memory, low clocks, and poor out-of-box software optimization.
"The Elbrus-8C server is very weak compared to Intel Xeon 'Cascade Lake'," said Anton Zhbankov, a representative for SberInfra, said at the Elbrus Partner Day conference (via ServerNews.ru) earlier this month. "Insufficient memory [256MB], slow memory, few cores, low frequency. Functional requirements not been met at all."
- Elbrus-8C: 8C/8T, 1.30 GHz, 16MB L3, 70W TDP, quad-channel DDR3-1600 memory, 28nm, 250 FP64 GFLOPS
- Intel's Xeon Gold 6230: Cascade Lake-SP, 20C/40T, 2.10 – 3.90 GHz, 27.5MB L3, 125W TDP, 14nm
- Elbrus-8CB: New microarchitecture, 8C/8T, 1.50 GHz, 16MB L3, 90W TDP, quad-channel DDR4-2400 memory, 28nm, 576 FP64 GFLOPS
In fact, SberInfra's evaluation was the first in-depth testing of the Elbrus-8C platform in a banking application. The evaluators compared dual- and quad-socket Elbrus-8C machines (16 - 32 cores per box) to a dual-processor server based on Intel's Xeon Gold 6230 processor that the company currently uses. SberInfra could not test the more powerful Elbrus-8CB as it is still not available despite being formally introduced.
…Cannot Compete Against Industry-Standard Parts
Being one of the largest banks in Europe that offers many more services than just banking, Sber has certain requirements for hardware and has its own test methodology for evaluating machines it considers deploying. This methodology includes the following:
- Functional Testing (44 parameters to make sure that a platform can run what Sber needs and can be managed how Sber needs it);
- Synthetic Testing (using PGbench from the PostreSQL suite as well as SPEC CPU 2017);
- Application Testing (using Java apps).
Every server begins from its chassis and some general features such as remote management, which Sber evaluates under its Functional Testing procedure. Apparently, an MCST Elbrus-8C machine failed 84% of Sber's Functional Testing as it could not be easily removed from the rack, lacked proper LED indicators, and came without remote management, which to a large degree made it unusable for usage in commercial datacenters. There is some hope, though.
"One of the surprising things about the Elbrus-8C server was that it is a real product," said Zhbankov. "It was a real server that we were given. […] It is an actual product that has its disadvantages, loads of disadvantages, but we can work with them."
Elbrus-8C Evaluation Summary
| Row 0 - Cell 0 | 4-way Elbrus-8C vs 2-way Intel Xeon Gold 6230 |
| SPEC CPU 2017 | 2.62 (base) ~ 3.15 (peak) times slower |
| PGbench/PostreSQL | 1.7 (read only profile) ~ 3.3 (read write profile) times lower |
| Java | 23 ~ 26 times higher response time |
The situation looks slightly better with the SPEC CPU 2017 benchmark, as the quad-chip Elbrus-8C was 2.62 (base) ~ 3.15 (peak) times slower than the dual Intel Xeon Gold 6230 machine, which was not that bad as SberInfra engineers expected a 20x to 30x difference. However, it should be noted that neither the x86 system nor the Elbrus machine achieved their peak performance numbers submitted by server makers to Spec.org.
Meanwhile, in PGbench/PostreSQL tests, the Xeon Gold 6230 machine was 1.7 (read-only profile) ~ 3.3 (read-write profile) times better (in terms of transactions per $100,000) than the Elbrus-8C server depending on the workload, which is significant but not dramatically lower.
With Java applications or emulated Java workloads, the situation got a lot direr for the Elbrus-8C platform that showed 23x to 26x times higher response time and did not meet any of Sber's quality-of-service requirements. According to the companies, the good news is that Java application startup times and response times improved with performance optimizations. The Elbrus-8C machine was still not quite competitive against the Xeon server, but its performance can be improved with software tweaks.
Will Take Years to Catch Up
But while SberInfra's engineers expected the Elbrus-8C machine to perform much worse and be orders of magnitude slower than Intel's Xeon Gold 6230 machine from 2019, even a two to three times performance difference is significant enough for commercial companies not to deploy a platform since it makes no financial sense.
"At the moment, Sberbank says no, we cannot deploy Elbrus machines into our ecosystem, but we are pleasantly surprised that it works at all," said Zhbankov.
For now, there are problems even with MCST's system design itself, so the CPU performance is something that Sber or any other hyperscaler would not even normally evaluate. Speaking of CPU performance, the company introduced the Elbrus-8CB several years ago and it is expected to arrive shortly. This chip promises considerably higher performance due to a new microarchitecture and improved memory support. Also, MCST has a rather ambitious server roadmap that includes a 12-core Elbrus processor, a 16-core CPU that was taped out last year, and even a 32-core system-on-chip for PetaFLOPS-class systems.
At some point in the future, MCST's Elbrus processors will get significantly faster than they are today. Still, the problem is that it takes the company an enormous amount of time to develop new CPUs and bring them to the market (e.g., the Elbrus-8CB was announced in 2018). Therefore, by the time the ambitious 32-core Elbruses arrive, chips from AMD and Intel will be orders of magnitude faster and more efficient in terms of performance-per-watt than they are today. This begs the question of whether various domestically designed Chinese or Russian CPUs will even catch up with those from leading developers.
The answer appears to be multifaceted. Companies like MCST can develop CPUs that are good enough for office workloads. Such systems may well be deployed by governments that can pay extra to support domestic CPU developers and not use foreign technologies. Homegrown CPUs can also be used to build supercomputers if things like scalability and energy efficiency are not a concern. However, it does not seem that such homegrown CPUs will catch up with developments from AMD, Intel, and emerging Arm-based server SoC designers any time soon.
