PCI Express 4.0 Brings 16 GT/s And At Least 300 Watts At The Slot (Update: Not Quite)

Update, 8/24/16, 2:06pm PT:PCI-SIG reached out to tell us that the power increase for PCI Express 4.0 will come from secondary connectors and not from the slot directly. They confirmed that we were initially told incorrect information. We have redacted a short passage from our original article that stated what were originally told, which is that the slot would provide at least 300W, and added clarification:

  • PCIe 3.0 max power capabilities: 75W from CEM + 225W from supplemental power connectors = 300W total
  • PCIe 4.0 max power capabilities: TBD

New value “P” = 75W from CEM + (P-75)W from supplemental power connectors.

To ensure optimum max power, it is recommended that card and system implementers refer to the associated PCI Express specifications that include high-power thermal and mechanical guidelines. Implementers should review also refer to the latest versions of the PCI Express x16 Graphics 150W-ATX Specification (including the supplemental power cable) and the PCI Express 225W/300W High Power Card Electromechanical Specification for additional information.

Original article:

The next iteration of PCI Express will double the bandwidth per lane, but that shouldn't surprise anyone. During Intel Developers Forum, we sat down with Richard Solomon, Vice President of PCI-SIG, and found the hidden gem in the next specification; it provides at least a 4x increase in power delivery at the slot, which might eliminate the need for auxiliary power cables with some GPUs.

The PCI-SIG consists of more than 730 paying members that provide input and a board of directors that help steer the process. The standards body is just "days away" from releasing version 0.7 of the PCIe Express 4.0 specification to members, according to Solomon. The final specification should come by the end of 2016, but the group will have an additional 0.9 step in between before completion at 1.0. Some companies, such as Mellanox, are shipping leading-edge PCIe products with the 0.7 spec, but will update when version 1.0 arrives. The group is already working on PCIe 5.0.

As we sat through our presentation with Salomon, I kept thinking back to the AMD Naples server motherboard the company displayed the previous night. AMD didn't want to divulge too many details about the next generation server platform, but a picture is worth a thousand words. If you look closely, there is a single 20-pin power connector like any other modern motherboard. There are also four 8-pin and two 6-pin power connectors. AMD marked the PCIe power connectors as follows:

P0 ABCD PWR
P1 ABCD PWR
P0 EFGH PWR
P1 EFGH PWR

These connectors are in addition to the processor 4+4 pin connector marked P0 and P1 CORE PWR. Even with a 64-core platform, that much power for the board and processors would severely limit AMD's chances of getting into data centers. We feel that AMD added the extra power for something else, like the PCIe edge connectors.

When we asked the PCI-SIG, we received the news that for the first time, PCIe will get a massive power increase at the connector. Solomon couldn't recall the exact ceiling because member companies have proposed several options. (We were originally told that it would be at least 300W, with a ceiling of 400-500W, but PCI-SIG has since disconfirmed that by stating that some of the power will come from secondary connectors.)

PCI Express 3.0 provides a maximum of 75 watts thus far, though many slots support the default 25W. The rating for a x16 lane card comes from 3 amps at 3.3 volts and 5.5 amps at 12 volts, which is a rating that dates back to the introduction of PCI Express 1.0a in 2003.

Speeds and Feeds

Swipe to scroll horizontally
PCIe VersionLine CodeTransfer Ratex1 Bandwidthx4x8x16
1.08b/10b2.5 GT/s250 MB/s1 GB/s2 GB/s4 GB/s
2.08b/10b5 GT/s500 MB/s2 GB/s4 GB/s8 GB/s
3.0128b/130b8 GT/s984.6 MB/s3.938 GB/s7.877 GB/s15.754 GB/s
4.0128b/130b16 GT/s1.969 GB/s7.877 GB/s15.754 GB/s31.508 GB/s

The next iteration of the PCI Express standard will officially pass the 1-gigabyte bandwidth mark and knock on the door of 2 gigabytes per second, per lane, per direction. The performance shifts from 3.0 to 4.0 while x16 devices like video cards get a massive headroom increase that will aid in providing the bandwidth needed for next generation virtual and augmented reality. PCI Express is a full-duplex protocol that can send and transfer data at the same time. The speeds listed above are for single direction performance. PCIe 4.0 uses the same encoding scheme as PCIe 3.0.

PCI Express 4.0 Usage

PCI Express has moved well beyond video cards. Currently, nearly every IO device routes through PCIe to send signals back to the CPU. We expect to see rapid adoption of the new standard for storage and networking products.

NVMe SSDs using PCIe 3.0 x4 recently reached the usable limit of the interface, such as the Samsung SM961 that reads data at 3,100 MB/s. Enterprise NVMe SSDs using the add-in card form factor and bifurcation will ship later this year with PCIe 3.0 x16. The increase in throughput will allow manufacturers to reduce costs by delivering the same performance with half the number of PCIe lanes. Analyst firm Forward Insights predicts enterprise PCIe storage devices will surpass the number of SATA products shipped between 2017 and 2018. 

At Computex 2016, Marvell told us to expect consumer networking products with 10-gigabit speeds in January (CES time frame). 10GbE will finally fall out of the enterprise sector and into the laps of us at home and provide roughly 1GB/s of data transfer speeds over commodity CAT 6 cabling. A single 10GbE connection will only require a single PCIe lane with the 4.0 specification.

Mobile, Internet of Things (IoT), and other battery-operated devices will benefit from low-power states and a new focus on burst performance. PCIe 4.0 adapts new L1 sub-states with half- and quarter-swing bursts, which use just 400 and 200-millivolt steps.

Connecting External Devices Through A Standardized PCIe Cable

During the last year, we've seen a number of companies build external graphics housings that route back to a small form factor PC via external cabling. The big fear is that a product from Company A will not work with a product from Company B. The PCI-SIG looks to end interoperability issues with the OCuLink standard. 

Storage systems will also adopt the PCIe-over-cable OCuLink connection. The cable is more graceful than SATA Express and the reworked SAS cables that we currently use for U.2 devices.

Chris Ramseyer
Chris Ramseyer is a Contributing Editor for Tom's Hardware US. He tests and reviews consumer storage.
  • ComputerSecurityGuy
    Most, not some. Of course the biggest issue is going to be that Intel will have to push even more bandwidth. And it won't show up until Cannonlake at the earliest.

    On another note, any word on HyperTransport 4.0?
    Reply
  • josejones
    I would like to know if PCIe 4.0 will be as pathetic as when PCIe 3.0 first came out and made absolutely no difference over PCIe 2.0 or, will PCI and its members get it right the first time?

    REMEMBER?:

    MSI Calls Out Gigabyte for "Not True PCIe 3.0"
    http://www.tomshardware.com/news/msi-gigabyte-pcie-gen3-mobo,13377.html

    Gigabyte Sets Record Straight on PCIe 3.0 Support
    http://www.tomshardware.com/news/gigabyte-msi-pcie-3.0-gen3-third-gen,13485.html

    And which CPU and mobo will PCIe 4.0 first support - the 200 or 300 series motherboard chipset? Will in be available in 2017 or 2018?

    We cannot omit just how bad PCIe 3.0 was when it first came out. Is it best to wait until the 2nd generation 4.0 or at least revision 2 or what?
    Reply
  • renz496
    18473353 said:
    I would like to know if PCIe 4.0 will be as pathetic as when PCIe 3.0 first came out and made absolutely no difference over PCIe 2.0 or, will PCI and its members get it right the first time?

    REMEMBER?:

    MSI Calls Out Gigabyte for "Not True PCIe 3.0"
    http://www.tomshardware.com/news/msi-gigabyte-pcie-gen3-mobo,13377.html

    Gigabyte Sets Record Straight on PCIe 3.0 Support
    http://www.tomshardware.com/news/gigabyte-msi-pcie-3.0-gen3-third-gen,13485.html

    And which CPU and mobo will PCIe 4.0 first support - the 200 or 300 series motherboard chipset? Will in be available in 2017 or 2018?

    We cannot omit just how bad PCIe 3.0 was when it first came out. Is it best to wait until the 2nd generation 4.0 or at least revision 2 or what?

    the performance will depend on application. back then is there any hardware that can really stress the usage of PCIE 2.0 making PCIE 3.0 as an absolute advantage to have?
    Reply
  • CyranD
    I believe 24 pin +4 Pin power connections on the motherboard currently maxes out at around 550 watts. Does that mean that motherboards that support pci-e 4.0 will require new/additional power connections since 550 watts would not be enough for multiple PCI-e 4.0 to use in a SLI/Crossfire setup for example.
    Reply
  • dstarr3
    Is there anything that's been close to saturating the bandwidth of PCIe 3.0? GPUs don't even saturate 2.0, and I don't think SSDs are that fast yet. I have to assume the biggest reason for the next generation is just the power delivery. Probably just doubled the bandwidth now because they'll need to eventually and we have the ability now.
    Reply
  • CRamseyer
    At Flash Memory Summit we saw a handful of products using 16-lane adapter cards with the lanes divided between 4x 4-lane cards (M.2). Storage is pushing the market but other areas like networking have a hand in the change. Mellanox has already displayed the ConnectX-5 with PCIe 4.0 support to drive dual 100GbE connections. The ConnectX-4 with PCIe 3.0 x16 can only drive around 108Gb/s throughput with two 100GbE connections running at the same time.
    Reply
  • Samer1970
    OMG ... 300 Watts ...

    How about compatibility with older cards? This is the first time they increase power.
    Reply
  • bit_user
    Nice writeup.

    I'm really excited about PCIe 4.0 (take that, NVLink!), but it comes too late for Intel's new server platform. So, we're probably looking at 2018, at the earliest, for support in Intel systems.
    Reply
  • SteelCity1981
    how did the samsung sm961 reach pci 3.0 x4 useable limits when its bandwith was 3,100mb's yet pci 3.0 x4 3.938gb's? there is still 838mb's unused.
    Reply
  • bit_user
    18473353 said:
    We cannot omit just how bad PCIe 3.0 was when it first came out. Is it best to wait until the 2nd generation 4.0 or at least revision 2 or what?
    This is a very selective reading of history, based on one incident with a bad board vendor. At launch, you could get SandyBridge-E boards that had true PCIe 3.0 support & full performance.

    But your question is still valid: which chipsets & CPUs will offer true PCIe 4.0 support? It's probably too early to say, but it definitely won't be in Skylake-E and likely not Kabylake.
    Reply