Are the M.2 sockets practically usable if running 4 Large GPUS

[danryanfl]

I am putting together a GPU machine for machine learning and I plan on putting 4 GTX 1080's into a new Z270 supermicro board (newegg listing).

Given that the M.2 slots sit behind the GPUs, and that I am going to have a wall of 10,000 compute cores working like crazy at times, should I avoid the M.2 slots and get more expensive U.2 drives that can be mounted away from the GPUs instead?

 

[greens]

That board makes little sense to me.
Aren't 6th/7th 1151 restricted to 16 PCIE lanes?
That would mean 4 4 4 4 for the GPUs, which really isn't enough for a GTX 1080.
PLUS you want to add MORE PCIE based devices? Isn't that going to throw system resources unavailable error?

Is there something i'm missing? Don't you need a 2011-v3 board to make this happen?

 

[Snipergod87]

You should check the manual, chances are plugging in that many cards will disable the M.2 and or U.2 slots

 

[mcnumpty23]

That board makes little sense to me.
Aren't 6th/7th 1151 restricted to 16 PCIE lanes?
That would mean 4 4 4 4 for the GPUs, which really isn't enough for a GTX 1080.
PLUS you want to add MORE PCIE based devices? Isn't that going to throw system resources unavailable error?

Is there something i'm missing? Don't you need a 2011-v3 board to make this happen?

think thats 16 pci-e lanes from the cpu and the rest from pch

so think 4 cards might all run at x8

though with out reading the motherboard manual thats a bit of guess work on my part

 

[danryanfl]

That board makes little sense to me.
Aren't 6th/7th 1151 restricted to 16 PCIE lanes?
That would mean 4 4 4 4 for the GPUs, which really isn't enough for a GTX 1080.
PLUS you want to add MORE PCIE based devices? Isn't that going to throw system resources unavailable error?

Is there something i'm missing? Don't you need a 2011-v3 board to make this happen?

 

[joex444]

Looks like it has a PLX chip, so it doesn't run x4/x4/x4/x4 but x8/x8/x8/x8.

Best advice is to check the manual to ensure that you can use all 4 PCIe slots and the M.2 port in PCIe 3.0 x4 mode at the same time. It will almost certainly disable something when you do that. I always read the manual before buying a board, to ensure it fits my needs. SuperMicro's website should have a PDF you can download.

What you'd be hoping for is that the M.2 slot drops the PCIe slots down to x8/x4/x8/x8 with the M.2 slot shared with the 2nd slot. (I mean, it could be any slot but that's what it would be in that case.) What you don't want is for it to drop to x8/NA/x8/x8 where the 2nd slot is disabled.

It looks like you are running a CUDA program or maybe cryptocurrency mining. You should really take a look at whether with four 1080 GPUs you can really get by with a Kabylake + Z270. There are plenty of X99 boards for under $300 that satisfy your need - M.2 PCIe 3.0 x4 + 4 PCIe 3.0 x4 or greater slots. Since the entry level LGA2011-3 CPUs have 28 PCIe lanes and you only need 20 lanes you should be fine with a 5820K or a 6800K. They're not much more expensive than a 7700K. Question is: does your application benefit more from high clock rate CPUs or more cores, or is it really not sensitive to CPUs at all?

In the latter case, your best bet might be an i3 CPU and a PLX enabled Z170/Z270 (Z270 having more PCIe lanes from the chipset). For clock speed, then a i7-7700K, and for cores, then X99.

 

[danryanfl]

The Z270 chipset has extra lanes for this purpose. According the manual you can have x8/x8/x8/x8 and still use the M.2 slots with no interference. Doesn't say anything about heat though of course.

 

[greens]

Well yeah the Z170 has extra lanes too but they are for SATA, I/O and stuff, they are different. I don't see how you can get extra PCIE lanes from the CPU... Can a multi GPU expert chime in on this? every build i've ever seen uses 2011-v3 for multi-GPU. Your CPU simply doesn't have the lanes.

 

[danryanfl]

The M.2 and U.2 sockets get their PCIE lanes from the PCH-H Z270. One M.2 shares with a U.2 and there is also some SATA sharing. It is on page 1-9 of the manual. The GPUs have lanes attached to the 1151 socket.

I think I am going to give this a try since the M.2 drives are so much cheaper. I will make sure to get GPUs that blow heat out the back and hope that is enough.

 

[greens]

Right...
"The GPUs have lanes attached to the 1151 socket."
16 of them.
16/4=4
4 lanes for a GTX 1080... not sure that is gonna cut it.

The m.2 drives are probably fine because they can also be used as SATA drives - which DO come from the mobo.

Your GPUs are still gonna be thirsty for bandwidth.

 

[TJ Hooker]

Right...
"The GPUs have lanes attached to the 1151 socket."
16 of them.
16/4=4
4 lanes for a GTX 1080... not sure that is gonna cut it.

The m.2 drives are probably fine because they can also be used as SATA drives - which DO come from the mobo.

Your GPUs are still gonna be thirsty for bandwidth.

Do you know the PCI bandwidth requirements for machine learning? For gaming a x4 connection would reduce performance of the cards, but I have no idea about machine learning.

 

[greens]

Not a clue. I don't know how those miner host machines with 10+ GPUs work either. I just know that 1151 doesn't have the lanes for anything beyond two GPU SLI/Crossfire. But, like the miners - he probably isn't going to be using SLI or crossfire, but rather Linux drivers.

 

[belo]

I have only seen that the m2 slots disable sata, and haven't seen yet where they disable pci-e. but I haven't looked closely at your board choice. I think what I might look for is a board with good m2 shielding for heat. note with 2 m2 slots a lot of boards leave you with only 2 sata slots left so you can do an optical and an extra hd if you need it and you're maxed.

x100 on reading the manual. before I bought my 270 I read the manual and it really clarifies some things you wont get out of the newegg listing.

 

[greens]

They don't need to disable - they are shared. If you don't have the PCIE lanes to the CPU, you can't use a PCIE based device in the M.2 slot. That is effectively disabled. Same with SATA - it shares bandwidth so if you use SATA X then it could disable m.2. Again this isn't so much a disabled as a "one or the other".

Shielding for heat? M.2 drives are hot little buggers - I would not advise any kind of shielding - they need air. Even having them behind the mobo is troubling if you don't have a nice cut out on your tray.

 

[belo]

They don't need to disable - they are shared. If you don't have the PCIE lanes to the CPU, you can't use a PCIE based device in the M.2 slot. That is effectively disabled. Same with SATA - it shares bandwidth so if you use SATA X then it could disable m.2. Again this isn't so much a disabled as a "one or the other".

Shielding for heat? M.2 drives are hot little buggers - I would not advise any kind of shielding - they need air. Even having them behind the mobo is troubling if you don't have a nice cut out on your tray.

Shielding reference is to some of the board builders buzz words for tougher components. Steel core, blah blah.