z170 chipset mining rig with more than 4 gpu

racminer · Dec 28, 2016

Hi

I would like to share my own experience with MSI Z170A pro carbone motherboard while building a mining rig for altcoins (ZEC, ETH, XMR).
First of all and in order to get it to accept USB 3.0 Powered PCI-E 1x -16x Risers, I have to mod the bios and set all PCIe lanes to gen1 (all 20 of them, I did this because I was unable to find which PCI socket goes with which pci lane).
Under window 10, I have succeeded in getting 4 GPU's to work fine (Rx480, R9 380 or R7 370). But I have not succeeded to run more than 4 x Rx480 gpus nor more than 4 x R9 380 gpus, I get the famous message "not sufficient resources ... code 12".

!!!!! However, 7 x R7 370 have worked just fine with no issues whatsoever !!!!

1) I don't think that the problem is a pci lanes assignment Chipset issue as is often reported, since 7x R7 370 has worked fine.
2) I don't think it is a IRQ issue as often reported also, because each R7 370 have had it own IRQ assigned.
3) I think that the problem is Memory Resource Assignment. By examining the control panel, I noticed that the R7 370 reserves much less memory and I/O port resources than the Rx480 or the R9 380.

Is it a driver issue? Is it possible to mod the driver in order to delete unneeded resources.

Does the chipset has anything to do with memory and i/o port assignment ??

****************
In any case, the real news here is that the MSI Z170A pro carbone works perfectly with 7 x Asus Strix R7 370 4Gb
****************

greens · Dec 28, 2016

Just throwing this out there, but could it be because the other cards require 4x lanes each to function?
Where the R7 370 is fine using 2?

You only get 16 lanes with z170, which means that with the R9 and RX you have 4+4+4+4=16 therefore, no more GPUs.
With the R7, it is probably happily running at x2, which places you at 2+2+2+2+2+2+2=14 so you have room for one more R7.

racminer · Dec 28, 2016

greens :

You may be right.
As far as seen in the bios, there are 20 PCI lanes. And all PCI slots are set to 1x (gen1).
Since I use 1x to 16x usr3.0 risers, I don't see how any gpu can run at 4x ?
Maybe I don't see quite well how all this work ! what I understand is that the GPU'a are physically connected to the mobo via 1x. I don't see how they could each acquire more that one lane ? :??:

greens · Dec 28, 2016

Actually, a full size GPU is set up for x16 lanes! That said, even a Titan x Pascal is perfectly happy with only x8 lanes though. This is what an SLI rig uses, 2 GPUs using 8 lanes each.

And in regards to the physical connection, you'll be surprised! The full size connection of a GPU is, like i said, equipped for 16 lanes.

Here is a wifi adapter that also uses PCIE but only 2 lanes! http://www.hwtools.net/jpg/MP2W_5300H_V212_1.jpg

See how short it is? It would still work in a full size slot though!

http://imagescdn.tweaktown.com/news/2/0/20242_03_intel_making_a_push_for_pci_express_x2_interface.png

You usually only ever see x2 and x16 on the boards though.

greens · Dec 28, 2016

Oh, and yes, Skylake CPUs have 20 lanes, but 4 of them are permanently assigned to I/O and are separate from the 16 PCIE lanes actually available to you.

greens · Dec 28, 2016

Bud, your post has actually gotten me curious, can i see the 3.1 to x16 adapater you got? This would be a neat way to make an mini ITX board with two GPUs in SLI!

racminer · Dec 28, 2016

greens :

Yes the risers are 1x and logicaly use only two lanes ...

https://miningcave.com/fr/produits/usb-3-0-pci-e-1x-16x-extender-riser-adapter/

greens · Dec 28, 2016

I see what you mean now! It definately looks and is advertised to only have PCIE 1x bandwidth.

And 4 RX/R9s work with 4 of those adapters?
I'm actually kind of surprised.

https://www.techpowerup.com/gpuz/
If you install that you can see how your GPUs are connected.
I'm curious what it says for BUS INTERFACE for your various setups.

Might provide some insight into how much bandwidth each card is actually getting.

TJ Hooker · Dec 28, 2016

@greens Skylake has 16 PCIe 3.0 lanes direct to the CPU (regardless of chipset), and another 20 through the Z170 PCH. Although the PCH-CPU link (DMI 3.0) is limited to 4 GB/s, so equivalent to PCIe 3.0 x4.

greens · Dec 28, 2016

Ohhh, i was a little confused about the lane assignments. Still correct in the result though - only 16 lanes available for GPUs and other devices.

TJ Hooker · Dec 28, 2016

You can run GPUs and other devices off of chipset PCIe lanes. For example, my mobo has 2 x16 (physical) slots. The first 1 is x16 direct from the CPU, the 2nd is x4 through the chipset. I could put a video card in either, or even both and crossfire.

greens · Dec 28, 2016

This I know to be incorrect.
Here is a diagram, those 20 PCIE are not available to the CPU as GPUs, that is where your SATA and I/O come in.
http://images.hardocp.com/images/articles/1439379258lUBczaKigM_1_1_l.png

You only get 16x PCIE lanes available in z170. This is true. You can use two GPUs in x8x8 SLI with z170. If you have an SLI set up, and you try to use a PCIE SSD, or wifi adapter, etc, with 2 GPUs in SLI you will get a resources error. period. (or your GPU will back down to x4 to free up lanes, choking its bandwidth)

racminer · Dec 28, 2016

greens :

This is what I get with TechPowerUpCpu-Z
Bus Interface: PCIe 3.0 @x1 1.1
As expected.

TJ Hooker · Dec 28, 2016

greens :

http://www.gigabyte.com/products/product-page.aspx?pid=5489#sp

I don't know what about that diagram makes you think the chipset lanes can't be use for graphics cards, but that's not right. The chipset provides a finite number of HSIO lanes. Some of these are dedicated as USB, Gigabit Eth, SATA. The rest can are flexible, and can be configured as generic PCIe lanes to different slots. So the Z170 chipset offers 'up to' 20 PCIe 3.0 lanes (as shown in the diagram you linked) in theory. For real life mobos, it will be less.
http://www.tomshardware.com/news/intel-100-series-hsio-chipset,30210.html

You're incorrect about not being able to use other PCIe devices without limiting GPU performance. I think you're conflating CPU PCIe lanes with chipset lanes. Yes, in some cases some slots/connectors may share bandwidth (e.g. the M.2 slot shares bandwidth with a PCIe slot), so you can' t use both at once. This will be explicitly stated on a mobo by mobo basis.
For example: if you're running SLI on an LGA 1151 board, you're running x8/x8 direct from the CPU; other slots that use PCIe lanes through the chipset will still work without affecting GPU bandwidth.

racminer · Dec 29, 2016

TJ Hooker :

greens :

http://www.gigabyte.com/products/product-page.aspx?pid=5489#sp

I don't know what about that diagram makes you think the chipset lanes can't be use for graphics cards, but that's not right. The chipset provides a finite number of HSIO lanes. Some of these are dedicated as USB, Gigabit Eth, SATA. The rest can are flexible, and can be configured as generic PCIe lanes to different slots. So the Z170 chipset offers 'up to' 20 PCIe 3.0 lanes (as shown in the diagram you linked) in theory. For real life mobos, it will be less.
http://www.tomshardware.com/news/intel-100-series-hsio-chipset,30210.html

You're incorrect about not being able to use other PCIe devices without limiting GPU performance. I think you're conflating CPU PCIe lanes with chipset lanes. Yes, in some cases some slots/connectors may share bandwidth (e.g. the M.2 slot shares bandwidth with a PCIe slot), so you can' t use both at once. This will be explicitly stated on a mobo by mobo basis.
For example: if you're running SLI on an LGA 1151 board, you're running x8/x8 direct from the CPU; other slots that use PCIe lanes through the chipset will still work without affecting GPU bandwidth.

All gpu's are connected to the mobo via 1x to 16x usb risers, if the issue were PCIe lanes, why would 7 x R7 370 work fine and not more than 4 R9 or Rx ?

Anyone could answer this?

racminer · Dec 30, 2016

I am finally able to get 6x Rx480 gpu's to work, the trick was to set TOLUD to 3Gb instead of Auto.

case closed.

neverfailedtofail · Jan 2, 2017

racminer :

hi,
i have exactly same problem with same mobo (MSI Z170A pro carbone) and 5 rx470s.
But I couldn't find how to change tolud?
can you describe it?
thanks in advance.

kngklla · May 26, 2017

neverfailedtofail :

did you ever find an answer to this? seems the old weebly article describing the process has been removed

thank you

z170 chipset mining rig with more than 4 gpu

Commendable

Splendid

Commendable

Splendid

Splendid

Splendid

Commendable

Splendid

Titan

Splendid

Titan

Splendid

Commendable

Titan

Commendable

Commendable

Commendable

Prominent

Share this page