HPE to Build Intel Sapphire Rapids-Based Crossroads Supercomputer for NNSA

Hewlett Packard Enterprise this week said that it had received a contract to build a supercomputer for the U.S. Department of Energy’s National Nuclear Security Administration (NNSA). The Crossroads supercomputer will be powered by Intel’s next-generation Xeon processors codenamed Sapphire Rapids and will be used to simulate nuclear weapons and guarantee the safety of the stockpile. Peak performance of the system is expected to exceed 80K TFLOPS.

The Crossroads Supercomputer

NNSA’s Crossroads supercomputer will use HPE’s Cray EX clustered architecture that use blades and liquid cooling. At present, Cray EX3000 systems use AMD’s Epyc 7002-based compute blades that include two 2P motherboards per blade (4 CPUs per blade) with up to 8 memory slots per socket that can handle up to 512 GB of memory using 64 GB DDR4-3200 modules.

The Crossroads system will be used to run high-resolution 3D simulations of nuclear weapons with full physics and geometric features. The supercomputer is said to leverage Intel’s Advanced Matrix Extensions (AMX) for AI acceleration as well as Sapphire Rapid’s memory architecture (which supports DDR5 SDRAM and uses a special accelerator for data copy and transformations) for these workloads. The DDR5 specification allows to build ultra-high-capacity memory modules running at high speeds, so it is logical to expect the upcoming blades for Crossroads to support more DRAM as well as a higher memory bandwidth than HPE’s Cray EX can support today. 

Rather surprisingly, NNSA does not disclose almost any technical details about its Crossroads supercomputer, such as the number of nodes or memory capacity. HPE says that Crossroads will be four times more powerful than the Trinity supercomputer used for the same kind of simulations today. Trinity offers a maximum Linpack performance of 20.1K TFLOPS as well as a theoretical peak performance of 41.5K TFLOPS, so it is reasonable to expect Crossroads to offer at least 80K TFLOPS of Linpack performance. 

The U.S. Department of Energy will pay $105 million for the supercomputer and expects HPE to deliver it in the spring of 2022. The system will be used by Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories from 2022 to 2026. 

NNSA and HPE stress that the Crossroads system will be delivered in early 2022, so by that time Intel will have to start volume production of its next-generation Sapphire Rapids CPUs that will also power the Aurora exascale-class supercomputer.

The Sapphire Rapids Platform

Intel’s Xeon ‘Sapphire Rapids’ is a processor that will bring a rather massive number of improvements to the company’s server platform.

Firstly, the CPU will be based on Intel’s next-generation microarchitecture (presumably Golden Cove) that will support Intel’s Advanced Matrix Extensions (AMX) as well as AVX512_BF16 and AVX512_VP2INTERSECT instructions that are particularly well suited for datacenter and supercomputer workloads. 

Secondly, the processor will feature an all-new DDR5-supporting memory controller and will be enhanced with Intel’s Data Streaming Accelerator (DSA), a data copy and transformation accelerator that offloads appropriate workloads from CPU. DSA will be useful for applications that process loads of data and intensively use persistent memory, high-performance storage, and networking operations.  

Thirdly, the Sapphire Rapids platform will support the PCIe Gen 5 bus with a 32 GT/s data transfer rate that is enhanced with CXL 1.1 protocol to optimize CPU-to-device (for accelerators) as well as CPU-to-memory (for memory expansion and storage devices) interconnects.  

Intel’s Sapphire Rapids will be made using the company’s 10nm Enhanced SuperFin process technology that will also be used to make codenamed Ice Lake-SP CPUs as well as upcoming Intel Xe-HP datacenter GPUs, so by the time Sapphire Rapids goes to mass production, the node promises to be rather mature.

Source: HPE