PCIe Gen3 Lanes Z77/Z87 chipset

Risaan

Honorable
Apr 27, 2013
20
0
10,520
my google skills have failed me :( so i hope you guys can help me :)
i am trying to look into the future.
intel 3570/3770/4570/4770 have 16 PCIe gen 3 lanes that means if i run SLI/CF i'll get 1x8 + 1x8 lanes.
1. from my understanding, gfx’s like gtx690/gtx titan and hd7990 will be bottleneck on PCIe gen2 x16 and PCIe gen3 x8 maybe more if you are looking on cuda/openGL ?
http://www.hardwarecanucks.com/forum/hardware-canucks-reviews/53901-nvidia-geforce-gtx-690-review-25.html
2. does that mean that we might have to use PLX when GFX’s like hd8xxx/hd9xxx and Gtx8xx/gtx9xx hits the market, when running SLI/CF ?
 
Solution
14% bottleneck:
The HD7970 was getting 86% of its maximum performance using the x4 interface. That's at 25% of the bandwidth of x16 (PCIe v3.0). That's not bad at all!

Intel CPU:
You have to realize that space on the CPU die is very, very important. They will increase PCIe bandwidth when it's warranted, but things like the iGPU are far more important. (speaking of which, Haswell has some really AMAZING power saving features, but it's uncertain how many will be in the desktop versions. For example, the ability to turn off and on really quickly so the average power is far lower than normal. Desktop? Probably. There's also a very cool feature where STATIC images for laptops and notebooks will stay in a small screen buffer so the screen...


As you see in the link, even a GTX690 barely saturates a PCIe v2 x16 interface. Bandwidth issues aren't a big deal.

You should note that many current AMD boards don't even have PCIe v3 yet. If you're building a new gaming rig and you're concerned about it, get a Haswell setup that supports 2xPCIe v3 x16. (I believe my Z77 Sabertooth gets 2xPCIe v3 x16 via PLX, otherwise it would be half that.)

Other:
- SLI/Crossfire still have micro-stutter issues
- SLI is overall superior to Crossfire
- the GTX700 series supposedly has improved frame rate metering to alleviate micro-stutter/frame time issues but we'll have to see.
 

Risaan

Honorable
Apr 27, 2013
20
0
10,520


Hi Photoboy thx for your reply :)

PCIe v2 x16 is not at big deal today, but it might be ind 2 years or before that.

haswell does not have more PCIe v3 lanes then ivy bridge so that will still be 2x PCIe v3 x16 with PLX right ?

Other issues
yeah i know about them. I belive that both nvidia and amd will improve that in the furture, more and more sites have access to frame capture system's. so we can better see what we get in a way. we might se not only a battle on the faste gfx, but also on the quality.
back to topic :)
so if i am buy'ing a z77/z87 motherboard with SLI/CF today, it needs PLX if i want to upgrade ind like 2 years to 2x new gfx's at this point. ?

i think that it suks tbh. you might need Ivy-bridge-E or maybe amd's steamroller if you want true PCIe gen3 x16 SLI/CF
 
Points:
1) HASWELL also has SIXTEEN CHANNELS of PCIe v3. For example, 2x (PCIe x8 v3.0).
I'm recommending it for other reasons.

2) PLX is on the motherboard. It's an option NOW and doubles the above bandwidth.
My Asus Z77 Sabertooth supports 2x (PCIe x16 v3). Thirty-two lanes.

I'm not sure what you perceive to be the issue?

There's also a point at which the current CPU or Memory become bottlenecks, regardless of how much theoretical bandwidth the motherboard has. Also, the PCIe bus is there for the CPU to send instructions to the GPU. With better software and improved GPU's, you'll see the GPU doing more calculations in proportion to the CPU instructions.

So a card that gets 2x the performance of a current card will likely use LESS than 2x of the PCIe bandwidth.

SUMMARY:
Long story short, it's good to investigate bottlenecks but it's essentially a non-issue when you get a motherboard with 32 lanes of PCIe v3. By the time this is an issue, it will make far more sense to build a new PC.
 

Risaan

Honorable
Apr 27, 2013
20
0
10,520


the issue i see is:
1. PLX gives a latency, this is in the region of 100 nanoseconds
2. PLX @ cost to motherboards, the cost might have been smaller on the CPU, with like 32 PCIe gen3 lanes.
3. PLX still only have 16 PCIe gen3 lanes to the CPU, so if you have 2 or more gfx's that use more then 16 gen3 lanes together, it wil still bottleneck. i belive that GFX's in the future will use more then 8 gen3 lanes each.

what i dont get is why Intel puts haswell on the market with only 16 gen3 PCIe lanes.
 


Good read here: http://www.anandtech.com/show/6170/four-multigpu-z77-boards-from-280350-plx-pex-8747-featuring-gigabyte-asrock-ecs-and-evga

1. I'm not sure what the LATENCY actually is, but I have PLX and perceive no latency issues. Do you realize how short 100 nanoseconds is? http://en.wikipedia.org/wiki/Nanosecond

2. COST.
Of course there's an added cost. However the only people for whom PCIe bandwith is even an issue would have to spend a lot on graphics cards and other components.

3. Your comment here is mostly true, and the issue is slightly complicated. Refer to the LINK above. However, it's not mainly about the number of lanes a card can use but about how FAST it requires them (bandwidth). It would be better to say "I think future cards will require more than 8GT/s or they will be bottlenecked." GT/s means Gigatransfers per second.

For example, I could put two SLOW cards in a system and there would be no bottlenecking issues at all even though they used all the available lanes.

If you haven't done so, start with Wikipedia:
http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.0

4. Why didn't Haswell use more than 16 lanes?
Because the market of people requiring more bandwidth than that is incredibly low.

*An HD7970 was tested with several games comparing PCIe v3 at x16 and at x4. I bet you'd think the HD7970 was HORRIBLY THROTTLED by the x4 interface right? Guess how much?

14%

OTHER:
Again, it's good to research things when buying a new PC, however for even high-end gamers it's going to be quite a while before PCIe bandwidth is an issue.

It's important to note as well that you can reduce your maximum bandwidth if you use some of the PCIe slots. A motherboard might have 16 available lanes, and if you only use a graphics card you have 16 still but adding ANY card at all might drop that to x8.

 

Risaan

Honorable
Apr 27, 2013
20
0
10,520
1. I'm not sure what the LATENCY actually is, but I have PLX and perceive no latency issues. Do you realize how short 100 nanoseconds is? http://en.wikipedia.org/wiki/Nanosecond
right it is a very slight issue. but still it is a issue that could have been avoided with like more PCIe gen3 lanes on the cpu

2. COST.
Of course there's an added cost. However the only people for whom PCIe bandwith is even an issue would have to spend a lot on graphics cards and other components.
true :)

3. Your comment here is mostly true, and the issue is slightly complicated. Refer to the LINK above. However, it's not mainly about the number of lanes a card can use but about how FAST it requires them (bandwidth). It would be better to say "I think future cards will require more than 8GT/s or they will be bottlenecked." GT/s means Gigatransfers per second.
okay :) i think future cards will require more then 8gt/s, i belive that gtx690 is just slight above that. so if you are looking to due SLI/CF in like 2 years it might be bottlenecking, because of the restrictions of 16gb/s from the cpu to the PCIe (cpu to PLX)

4. Why didn't Haswell use more than 16 lanes?
Because the market of people requiring more bandwidth than that is incredibly low.
you might be right here. but how does that look in like 2 years ?

*An HD7970 was tested with several games comparing PCIe v3 at x16 and at x4. I bet you'd think the HD7970 was HORRIBLY THROTTLED by the x4 interface right? Guess how much?

14%
the hd7970 was not doing 4gb/s all the time. i got it :) and the same will go for new cards.
a bottleneck of 14% is still bad in my world :(

OTHER:
Again, it's good to research things when buying a new PC, however for even high-end gamers it's going to be quite a while before PCIe bandwidth is an issue.
i am still not sure about that tbh :( but i guess only time will tell.

It's important to note as well that you can reduce your maximum bandwidth if you use some of the PCIe slots. A motherboard might have 16 available lanes, and if you only use a graphics card you have 16 still but adding ANY card at all might drop that to x8.
Spot on :)

many thanks for your time! you have helped me alot.
 
14% bottleneck:
The HD7970 was getting 86% of its maximum performance using the x4 interface. That's at 25% of the bandwidth of x16 (PCIe v3.0). That's not bad at all!

Intel CPU:
You have to realize that space on the CPU die is very, very important. They will increase PCIe bandwidth when it's warranted, but things like the iGPU are far more important. (speaking of which, Haswell has some really AMAZING power saving features, but it's uncertain how many will be in the desktop versions. For example, the ability to turn off and on really quickly so the average power is far lower than normal. Desktop? Probably. There's also a very cool feature where STATIC images for laptops and notebooks will stay in a small screen buffer so the screen updates via this attached buffer and MOST of the remaining computer is shut off. Desktop version? Unlikely. In the laptop/mobile Haswell devices it's apparently possible to get below 1Watt not counting the screen!)

If an HD7970 can achieve 86% of its performance on 25% of the CPU PCIe bandwidth you can see why increasing that bandwidth is pretty low on its list of priorities. However, it's my understanding that COMPUTE functions (i.e. OpenCL) can use much more PCIe bandwidth than a game would. I believe an HD7970 basically maxes out the PCIe v2.0 x8 interface. That's HALF of what the latest Intel CPU's provide.

I bought an HD3870 partially because it was supposed to have awesome video conversion tools; they've said that every card since... at least current HD7000's have great Compute capabilities so we just need great OpenCL software! (When the HD7970 came out I went "awesome" then remembered that it Compute only matters if the software exists AND I need that software to utilize it.)

Off topic:
If you do any Converting of video or remove subtitles, audio etc from video here's two awesome programs:
1. Handbrake
2. mkvtoolnix

I've tried all the main video converting software including MediaEspresso (it sucks big time) and Handbrake is just amazing (and free). I use Faststone as my picture viewer and Musicbee for audio.
 
Solution

Risaan

Honorable
Apr 27, 2013
20
0
10,520
I can now see where i am wrong :)
a single gfx will probly not be peaking more then 4gb/s within 2 years, when a hd7970/gtx680 is peaking around 2gb/s today.
and the GTX titan that is around 47% faster then the gtx680, will not use 47% more bandwidth (3gb/s) as i understand it.
i hope i got that right :)

Again many thanks for your help.
 


The ratio of GPU calculations vs PCIe bandwidth (or GPU vs CPU scaling software demands) is one reason. The MAIN reason is simply that the PCIe bandwidth requirement doesn't increase as quickly as some might think. And let's be clear, when it becomes essential, that's when a new PCIe version (i.e. PCIe v4.0 is released).

*Other:
I don't want to appear rude, but I recommend you pay more attention to your terminology. With reference to this link:
http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.0

Note:
1) You can use GB/second or GT/sec however understand the differences between them.

2) You say "gbps" which means Gigabits per Second; a BYTE is composed of eight BITS so if you say "gbps" and mean "GB/s" you're off by 8x the value you meant. Not only did you likely mean "GB/s" (Gigabytes per Second) but I'm not sure where you got the "2gps" value anyway since it's not accurate in either case. A 16-slot PCIe v3.0 interface can handle a bandwidth of 15.75 GB/ and a GTX680 would use just under HALF of that (so roughly 7GB/s).

This is getting picky, but you say "gfx" instead of saying "graphics card"; while I know what you mean the more mistakes there are the harder it is to read a comment. It doesn't mean graphics card:
http://en.wiktionary.org/wiki/GFX

3) Bandwidth is very tricky to discuss:
- Do we mean PCIe bandwidth and bottlenecks?
- Are we talking about the GPU to Video RAM (on the graphics card) only?
- Are we talking theoretical maximums? (for example the "60MB/sec" number you constantly see for USB2 devices such as card readers despite the fact that the device itself isn't that fast.

OTHER:
multi-GPU:
SLI/Crossfire use a feature called "AFR" or Alternate Frame Rendering. Essentially the same data exists in the Video RAM of each set of VRAM and each GPU works on it. If that's the case, would a GTX690 (two GPU's on one card) require DOUBLE the PCIe bandwidth of a single GPU, the SAME as a single GPU or somewhere in between? I leave that up to you to research if you wish.

(Also, is there a DIFFERENCE in PCIe bandwidth usage between two SEPARATE cards versus two identical GPU's on the SAME card?)

Summary:
PCIe bandwidth isn't a huge concern. CPU's and motherboards that support more PCIe bandwidth are released as the demand warrants it.
Cheers.