Netflix's data center engineering team took on a mission recently to double the encrypted video bandwidth of the video streaming company from 100 Gbps to about 200 Gbps. The company reached its goal on both Intel and AMD-powered servers, with the AMD Epyc servers seeing triple the bandwidth compared to before.
Intel Xeon Server Improvements
As detailed by Netflix senior software engineer Drew Gallatin at the EuroBSDcon 2019 conference and detailed by Phoronix, Netflix was serving 100 Gbps of encrypted video data from a single server since 2016 using Intel Broadwell Xeon CPUs to power its FreeBSD servers. The Broadwell CPUs feature a memory bandwidth of 60 GBps and 40 PCIe 3.0 lanes supporting 32 GBps of I/O bandwidth.
The company’s newer server processors include Intel Skylake and Cascade Lake with 90 GBps memory bandwidth and 48 PCIe 3.0 lanes with support for 38 GBps I/O bandwidth. But in order to get to its encrypted video bandwidth goal of 200 Gbps, Netflix needed at least 64 PCIe 3.0 lanes.
How could Netflix achieve that? The answer was to “throw another CPU socket” at the problem, according to Gallatin's presentation, doubling the CPU sockets per server.
As a result, Netflix's Intel server went from 105 Gbps to 191 Gbps. The new dual-socket Intel-based server has a 180 GBps memory bandwidth, 96 PCIe 3.0 lanes and about 75 GBps of /IO bandwidth.
AMD Epyc Server Improvements
Netflix also uses AMD's "Rome” Epyc server chips for encrypted video data. With AMD’s chips, there wasn’t a need to do dual-socket, as Netflix could use a “quad chiplet” Epyc CPU per server. The AMD CPUs support 120-150 GBps memory bandwidth, 128 PCIe 3.0 / PCIe 4.0 lanes with 100 GBps I/O bandwidth (200 GBps for PCIe 4.0).
AMD CPUs started out significantly slower than the Intel counterparts, serving only 68 Gbps of encrypted video data per server. This was likely due to FreeBSD not having as many optimizations in place for the new Zen architecture and Epyc server chip platform.
Once the Netflix team applied all the optimizations they could, the single-socket Epyc server was able to catch-up and even slightly surpass the dual-socket Xeon server, tripling its initial performance from 68 Gbps to 194 Gbps (compared to an ultimate 191 Gbps for Xeon).
The Netflix team did criticize AMD for not yet having well-developed tools for server operating systems, such as Linux. However, with the rise in popularity of AMD’s Epyc server chip platform, we can expect AMD to invest much more aggressively in the development of its tools.
