Am I using too many PCIe lanes?

26 I/O ports TOTAL, 6 are dedicated to USB 3.0 (max 10), leaving 20 ports which can be USB 3.0, SATA Revision 3, or PCI-E 3.0. After subtracting 16 lanes for x8/x8 PCIe 3.0 CF or SLI configurations, there are 4 lanes remaining.



There is also a maximum of 6x SATA6Gb/s ports that can be used out of the 20 I/O ports left after subtracting the 6x USB 3.0. Many people don't know that every PCH has PCI-E lanes, and they are used for connections to other devices such as NICs and extra USB or SATA controllers, and so people assumed that because they saw 20x PCI-E 3.0 it must be referring to the CPU's PCI-E controller, but actually it's referring to the PCH's PCI-E controller.

Click to expand...

To more precisely answer your question, Z170 has 20 PCI-E lanes at max independent of the ones hosted by the CPU. The CPU and PCH connect via a DMI 3.0 interface on all of the boards with the exception of the H110 board, that uses the older DMI 2.0 interface. This is equevilant to roughly a PCI-E 3.0 x4 connection, but it does not consume any PCI-E lanes from the CPU or motherboard.

The chipset uses what are known as HSIO lanes, which are essentially the same as PCI-E 3.0 lanes to connect to virtually everything else. A good example of what is going on here is to compare it to graphics cards. In the past, like the Voodoo 2, there were a ton of different components and fixed function units used to handle the various tasks. Eventually, most of these pieces were replaced when graphics cards moved away from fixed function units to general use shaders (I.E. CUDA/Stream Processors/Pixel Shaders). Chipsets are experiencing much the same thing, as Intel moves away from creating a chipset that contains controllers for SATA-III, SATA-II, USB 3.0, USB 2.0, PCI-E etc. to a multi-purpose controller unit capable of running all of these.

Z170 chipset has 26 of these, six of which are hard coded to be USB 3.0, with an additional 4 that can be configured as USB 3.0. The USB 2.0 slots do not consume any HSIO lanes. Of the 26, if all potential USB 3.0 lanes are used that leaves us with 16 HSIO lanes remaining (or 20 at most).

Z170 also supports up to 6 SATA-III connections that also consume one HSIO lane each. If all of these are configured and in use, then the motherboard now only has 10 HSIO lanes remaining (or 20 at most assuming no SATA points and no additional USB 3.0 above the initial six.)

You drop an additional HSIO lane for a single LAN NIC, leaving 9 HSIO lanes.

Of these remaining 9 lanes, all of them are essentially PCI-E 3.0 lanes, and can be configured into x1, x2 or x4 configurations. Any additional on-board features such as extra USB 3.0 or 3.1 slots, additional SATA ports, or secondary networking devices further reduce this number by at least one.

So, long story short, Z170 has at most 20 PCI-E 3.0 lanes available, if the motherboard OEM decides to not configure any HSIO lanes for anything and sells you a motherboard without any SATA ports, additional USB 3.0 ports, networking capabilities, or any other additional similar features.

As no motherboard OEM is going to do that, however, you likely have 9 PCI-E 3.0 lanes or less on a Z170 board. Often, you will see motherboards that have far too much connectivity like this one from ASRock:
http://www.tomshardware.com/news/asrock-fatal1tyz170-pr...

In which case, the motherboard has several of the connectivity interfaces shared with other devices, and just about anything you connect to the motherboard disables something else.As for the 16 lanes coming from the CPU, those are still reserved for GPUs typically.

Click to expand...

Z170 Platform including CPU = Max 42 PCI-E 3.0 lanes (typically 17 shared with other devices like USB, SATA, NIC, 16 reserved for GPU)

Z170 chipset = 26 PCI-E 3.0 lanes, can be segmented in groups of x1/x2/x4 (17 typically shared with other devices like USB, SATA, NIC)

Intel CPU = 16 PCI-E 3.0 lanes, can be segmented in groups of x4/x8/x16 (almost exclusively used for GPU)

The thing with Skylake chipsets is that explaining all of this is a mouthful and hard to get across. In the article I linked there is an image of the chipsets HSIO lanes that diagrams how they can be segmented, what lanes are shared with what, etc. Study that, it should help to clarify what I am attempting to say.

The reason we cannot have x16/x16 multi-GPU configurations, is because the chipset is incapable of supporting more than an x4 connection to any device that connects directly to it. Only the CPU is capable of that, which bypasses the chipset entirely. This is mostly because the connection between the CPU and chipset is essentially limited to a PCI-E 3.0 x4 connection, and so anything greater than an x4 connection would be bottlenecked by the DMI 3.0 interconnect between the CPU and chipset.

This is not to say that if you have an M.2 SSD with a PCI-E 3.0 x4 connection and a GPU connected to the chipset via a PCI-E 3.0 x4 connection that you will necessarily have a bottleneck, as it will alternate between the two devices when transferring data, but it may have some negative side effects (increased latency).

If you are wanting to have a dual-GPU and dual-M.2 setup, you should be fine with most Z170 motherboards (you need to check them carefully), but if you were planning to add six SATA drives to that, you would probably end up disabling one of the M.2 ports.

-Michael Justin Allen Sexton

Click to expand...