How Many PCIe Lanes Do You Need?
We've already proven that the performance impact of reducing PCIe bus bandwidth on a single card is essentially negligible (see The Myths Of Graphics Card Performance: Debunked, Part 2). But does the same claim hold true in multi-GPU configurations?
A while back, I was struggling with what seemed like poor performance scaling from GeForce GTX 980s in SLI. I was seeing numbers 30% lower than what I was expecting. And my CPU wasn't the bottleneck, either. Furthermore, the issue only materialized when I had a G-Sync-capable display connected, regardless of whether G-Sync was on or off. I came to suspect some sort of driver issue, before Nvidia and EVGA forum users suggested that I was the only one having this particular issue. If you're a system builder, this is unambiguous guidance: you are the one doing something wrong!
I couldn't make sense of what I was seeing until, about a week later, I remembered that I had left the PCIe Generation setting in my motherboard's firmware at first-gen transfer rates. I was running the two cards at x8 PCIe 1.0! That's a paltry 2 GB/s bandwidth, equivalent more or less to two third-gen lanes. ASRock has a nifty setting in its UEFI that lets you specify this parameter. It's something you never want to mess with unless you are, as I was, testing different PCIe link speeds.
Switching back to PCIe 3.0 in the motherboard UEFI alleviated that 30% performance handicap. But, perhaps even more interestingly, it allowed me to make some interesting inferences.
- Eight lanes (x8) of PCIe 3.0 is more than enough for two top-tier Nvidia cards in SLI. That is, from a PCIe bus effectiveness standpoint, you won't benefit materially from a motherboard with an (expensive) PLX chip or the leap to Intel's LGA 2011-v3 interface.
- Four lanes (x4) of PCIe 3.0 would most likely be fine for SLI. Nvidia does not support three-way SLI on x8/x4/x4-capable PCIe 3.0 motherboards. That's an uncommon scenario, so it matters little anyway. Three-way SLI is pretty rare to start, and if you're in the market for $1500 worth of graphics cards, you can most likely afford an extra $100 for a PLX bridge-equipped motherboard that'll give you the number of lanes Nvidia requires.
- G-Sync somehow materially increases usage of the PCIe bus in SLI. This isn't an issue per se, but it's a somewhat interesting fact if you are curious about understanding precisely how new technologies work. I still wonder why this is the case.
