Sign in with
Sign up | Sign in
Your question

AMD to integrate PCIe into CPU

Last response: in CPUs
Share
Anonymous
a b à CPUs
July 20, 2005 11:58:35 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

AMD to integrate PCIe
http://www.theinquirer.net/?article=24756

Yousuf Khan

More about : amd integrate pcie cpu

Anonymous
a b à CPUs
July 20, 2005 3:25:03 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Wireless, sound, IDE/SATA, USB/Firewire. Too many features that just
don't deserve a piece of the CPU real-estate, and don't really need the
full speed of the CPU to be dedicated to them. Just a CPU and a
southbridge basically. Just one step removed from a SOC.

Other things I see them possibly using the integrated PCI-e connector
for is integrated shared memory video. They can use the PCI-e video
protocols to share memory with the integrated video chipset directly.

Another use would be to offer even faster full dual-x16 SLI/Crossfire
support. They can connect one high-end video card to a northbridge x16
connector while the other one uses the CPU's x16 connector.

On the server front, they can connect things like commodity PCI-e
Infiniband cards directly to the CPU for HPC clusters.

Yousuf Khan
Anonymous
a b à CPUs
July 20, 2005 3:51:57 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

YKhan wrote:
>
> On the server front, they can connect things like commodity PCI-e
> Infiniband cards directly to the CPU for HPC clusters.
>
LOM..."Landed-on-motherboard" more likely.

RM
Related resources
Anonymous
a b à CPUs
July 20, 2005 7:54:43 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Robert Myers wrote:
> YKhan wrote:
> >
> > On the server front, they can connect things like commodity PCI-e
> > Infiniband cards directly to the CPU for HPC clusters.
> >
> LOM..."Landed-on-motherboard" more likely.

No, that is already done now, through Hypertransport. But built-in
Infiniband would be a very specialized requirement. That would make it
a very specialized subcategory of an already specialized subcategory.
Can't see the economies of scale being all that good for a motherboard
with built-in Infiniband. This way they can plug a bog-standard PCIe
Infiniband adapter (as bog-standard as those things get anyways), and
get slightly better latency out of it. May not be as good as
motherboard Infiniband, but better than through a chipset PCIe
connector.

Yousuf Khan
Anonymous
a b à CPUs
July 20, 2005 10:12:37 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Wed, 20 Jul 2005 07:58:35 -0700, YKhan wrote:

> AMD to integrate PCIe
> http://www.theinquirer.net/?article=24756
>
> Yousuf Khan

Two thoughts occur first that motherboards are about to get a fair bit
cheaper and second that overclocking is about to get more complex.

sounds like motherboards are basicly going to get turned into sockets and
a few things that wouldn't work inside the cpu. im sure ive read somewhere
that wireless and sound will need to remain seperate due to the way they
work.
July 22, 2005 8:35:56 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Wed, 20 Jul 2005 18:12:37 GMT, epaton <epaton@null.com> wrote:

>On Wed, 20 Jul 2005 07:58:35 -0700, YKhan wrote:
>
>> AMD to integrate PCIe
>> http://www.theinquirer.net/?article=24756
>>
>> Yousuf Khan
>
>Two thoughts occur first that motherboards are about to get a fair bit
>cheaper and second that overclocking is about to get more complex.
>
>sounds like motherboards are basicly going to get turned into sockets and
>a few things that wouldn't work inside the cpu. im sure ive read somewhere
>that wireless and sound will need to remain seperate due to the way they
>work.
>
Poor VIA, SIS, and ULI - they will be relegated to making commodity
south bridges, or fight mighty Intel for a piece of Pentium chipset
market. Nvidia and ATI have at least something to fall back on -
graphics. High end GPU will stay separate from CPU at least for quite
a while. OTOH, low end GPU may find its way into south bridges,
making them a bit less of a cheap commodity.
Anonymous
a b à CPUs
July 22, 2005 12:38:58 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

nobody@nowhere.net wrote:
> Poor VIA, SIS, and ULI - they will be relegated to making commodity
> south bridges, or fight mighty Intel for a piece of Pentium chipset
> market. Nvidia and ATI have at least something to fall back on -
> graphics. High end GPU will stay separate from CPU at least for quite
> a while. OTOH, low end GPU may find its way into south bridges,
> making them a bit less of a cheap commodity.
>

Considering the cooling requirements of even a low-end GPU (cooling fins
coming out all over the place), it's unlikely that they'll try to
integrate the GPU with the southbridge. The video chip overheats and you
lose connection to your hard drives? :-)

Yousuf Khan
July 23, 2005 3:08:10 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Fri, 22 Jul 2005 08:38:58 -0400, Yousuf Khan <bbbl67@ezrs.com>
wrote:

>nobody@nowhere.net wrote:
>> Poor VIA, SIS, and ULI - they will be relegated to making commodity
>> south bridges, or fight mighty Intel for a piece of Pentium chipset
>> market. Nvidia and ATI have at least something to fall back on -
>> graphics. High end GPU will stay separate from CPU at least for quite
>> a while. OTOH, low end GPU may find its way into south bridges,
>> making them a bit less of a cheap commodity.
>>
>
>Considering the cooling requirements of even a low-end GPU (cooling fins
>coming out all over the place), it's unlikely that they'll try to
>integrate the GPU with the southbridge. The video chip overheats and you
>lose connection to your hard drives? :-)
>
> Yousuf Khan

Low end GPU like X300 can do with passive heatsink, and quite a few
north bridges now need a fan even without graphics. So they'll slap a
fan on the south bridge/GPU combo. If a fan is not enough a BIG fan
will do. After all they'll need to sell something, and the market for
cheap integrated chipsets will always be there. Looks like nobody at
Intel is afraid to lose connection to RAM because the integrated
Extreme Graphics could overheat ;-)
Anonymous
a b à CPUs
July 23, 2005 5:39:13 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

nobody@nowhere.net wrote:
> Low end GPU like X300 can do with passive heatsink, and quite a few
> north bridges now need a fan even without graphics. So they'll slap a
> fan on the south bridge/GPU combo. If a fan is not enough a BIG fan
> will do. After all they'll need to sell something, and the market for
> cheap integrated chipsets will always be there. Looks like nobody at
> Intel is afraid to lose connection to RAM because the integrated
> Extreme Graphics could overheat ;-)
>

One thing nobody has mentioned yet is the shear irony of this situation.
Intel created PCI-e as a competitor to Hypertransport, because they
refused to adhere to a standard that AMD came up with. AMD gave the
green light to PCI-e without even a fight, knowing full well that PCI-e
and HT would be compatible with each other (just slightly different
physical layers), and now it may come up with the first PCI-e integrated
into the CPU.

Yousuf Khan
Anonymous
a b à CPUs
July 23, 2005 7:27:32 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Yousuf Khan wrote:
> nobody@nowhere.net wrote:
>
>> Low end GPU like X300 can do with passive heatsink, and quite a few
>> north bridges now need a fan even without graphics. So they'll slap a
>> fan on the south bridge/GPU combo. If a fan is not enough a BIG fan
>> will do. After all they'll need to sell something, and the market for
>> cheap integrated chipsets will always be there. Looks like nobody at
>> Intel is afraid to lose connection to RAM because the integrated
>> Extreme Graphics could overheat ;-)
>>
>
> One thing nobody has mentioned yet is the shear irony of this situation.
> Intel created PCI-e as a competitor to Hypertransport, because they
> refused to adhere to a standard that AMD came up with. AMD gave the
> green light to PCI-e without even a fight, knowing full well that PCI-e
> and HT would be compatible with each other (just slightly different
> physical layers), and now it may come up with the first PCI-e integrated
> into the CPU.
>
> Yousuf Khan

Sigh. where do you guys get these fairy stories? PCI-E was invented as
an IO expansion network to replace pci-x which was reaching the end of
its rope and took too many pins. InfiniBand was too server oriented.

Is everybody in this group full of conspiracy theories? I am really
starting to wonder about you guys.

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 24, 2005 4:59:15 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

YKhan wrote:
> Perhaps this will remind you?
>
> Approval near on Intel PC-overhaul plan | CNET News.com
> http://news.com.com/2100-1001-270823.html
>
> Yousuf Khan
>

Yes, it says intel got pci-e adopted. Hypertransport is totally
different thing, capable of driving a few inches. It is a FSB. Why the
doof that wrote the article even mentioned it isn't clear.



--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 24, 2005 10:29:14 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Del Cecchi wrote:
> YKhan wrote:
>
>> Perhaps this will remind you?
>>
>> Approval near on Intel PC-overhaul plan | CNET News.com
>> http://news.com.com/2100-1001-270823.html
>>
>> Yousuf Khan
>>
>
> Yes, it says intel got pci-e adopted. Hypertransport is totally
> different thing, capable of driving a few inches. It is a FSB. Why the
> doof that wrote the article even mentioned it isn't clear.
>

Because there was a time when HT was proposed as the next generation
PCI. It was initially going to allow PCI to get faster by simply
splitting each PCI slot into its own PCI bus, with each of the PCI buses
connected over HT. Then eventually they were talking about HT gaining
its own slot connector and people using HT directly.

Both of those scenarios actually did come true, in a way. HT has become
a very popular underlying layer for PCI, PCI-X and even PCI-E. There is
also a slot connector standard for HT called HTX, but it's not
necessarily all that popular.

Yousuf Khan
Anonymous
a b à CPUs
July 26, 2005 5:43:01 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Sun, 24 Jul 2005 18:29:14 -0400, Yousuf Khan <bbbl67@ezrs.com>
wrote:
>Del Cecchi wrote:
>> Yes, it says intel got pci-e adopted. Hypertransport is totally
>> different thing, capable of driving a few inches. It is a FSB. Why the
>> doof that wrote the article even mentioned it isn't clear.

There's no spec that shows exactly how far each could be driven, but I
suspect that you'll find Hypertransport and PCI-Express could achieve
comparable distances for similar data rates. My idea of "a few
inches" in computer designs is 2-3", and there are definitely HT
setups running at high data rates that go further than that (I would
guess that the furthest I've seen would be about 12" for a 16-bit,
2000MT/s link).

>Because there was a time when HT was proposed as the next generation
>PCI. It was initially going to allow PCI to get faster by simply
>splitting each PCI slot into its own PCI bus, with each of the PCI buses
>connected over HT. Then eventually they were talking about HT gaining
>its own slot connector and people using HT directly.
>
>Both of those scenarios actually did come true, in a way. HT has become
>a very popular underlying layer for PCI, PCI-X and even PCI-E. There is
>also a slot connector standard for HT called HTX, but it's not
>necessarily all that popular.

To the best of my knowledge there is only ONE HTX add-in card, an
Infiniband card from Pathscale. This card was recently used to set
some world records for low-latency communication in clusters.

The slot is actually VERY similar to PCI-Express (same physical
connectors) and the specs are designed to make it easy to have both
PCI-E and HTX on the same board.

Really when you get right down to it, Hypertransport and PCI-Express
started out with rather different goals but the end result is
surprisingly similar. I guess there really are only so many ways to
skin a cat.

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca
Anonymous
a b à CPUs
July 26, 2005 3:05:21 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Yousuf Khan wrote:
> Del Cecchi wrote:
> > HT can go maybe a foot, if you are really lucky. Work out the skew
> > budgets. At 2000 MT, the board is allocated less than 100 ps as I recall.
> >
> > PCI-E on the other hand can go several meters.
> >
> > Totally different approaches.
>
> Which is why PCI-e never got adopted as a CPU to CPU interconnect.
>
> Yousuf Khan

One never knows what the future holds. Anyway, it's pretty obvious
that parallel transmission (read HT) is the way of the past. If you
look at any high performance interconnect, they are all serial. Talk
to the Rambus guys, they know what they are doing...

Now, as to whether serial connections between CPUs is a good idea, I am
not entirely sure; I suspect Del is far more qualified to discuss that
topic than I am. Generally, serial connections can be driven far
faster, but there is slighly longer latency for SERDES.

HT was never envisioned to replace PCI-X, PCI or anything else.
Yousuf, you should at least try to distinguish yourself from AMD PR
personnel...

David
Anonymous
a b à CPUs
July 26, 2005 3:07:29 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Tony Hill wrote:
> On Sun, 24 Jul 2005 18:29:14 -0400, Yousuf Khan <bbbl67@ezrs.com>
> wrote:
>
>>Del Cecchi wrote:
>>
>>>Yes, it says intel got pci-e adopted. Hypertransport is totally
>>>different thing, capable of driving a few inches. It is a FSB. Why the
>>>doof that wrote the article even mentioned it isn't clear.
>
>
> There's no spec that shows exactly how far each could be driven, but I
> suspect that you'll find Hypertransport and PCI-Express could achieve
> comparable distances for similar data rates. My idea of "a few
> inches" in computer designs is 2-3", and there are definitely HT
> setups running at high data rates that go further than that (I would
> guess that the furthest I've seen would be about 12" for a 16-bit,
> 2000MT/s link).
>
>
>>Because there was a time when HT was proposed as the next generation
>>PCI. It was initially going to allow PCI to get faster by simply
>>splitting each PCI slot into its own PCI bus, with each of the PCI buses
>>connected over HT. Then eventually they were talking about HT gaining
>>its own slot connector and people using HT directly.
>>
>>Both of those scenarios actually did come true, in a way. HT has become
>>a very popular underlying layer for PCI, PCI-X and even PCI-E. There is
>>also a slot connector standard for HT called HTX, but it's not
>>necessarily all that popular.
>
>
> To the best of my knowledge there is only ONE HTX add-in card, an
> Infiniband card from Pathscale. This card was recently used to set
> some world records for low-latency communication in clusters.
>
> The slot is actually VERY similar to PCI-Express (same physical
> connectors) and the specs are designed to make it easy to have both
> PCI-E and HTX on the same board.
>
> Really when you get right down to it, Hypertransport and PCI-Express
> started out with rather different goals but the end result is
> surprisingly similar. I guess there really are only so many ways to
> skin a cat.
>
> -------------
> Tony Hill
> hilla <underscore> 20 <at> yahoo <dot> ca


HT can go maybe a foot, if you are really lucky. Work out the skew
budgets. At 2000 MT, the board is allocated less than 100 ps as I recall.

PCI-E on the other hand can go several meters.

Totally different approaches.

del

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 26, 2005 4:50:03 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Tony Hill wrote:
>>Both of those scenarios actually did come true, in a way. HT has become
>>a very popular underlying layer for PCI, PCI-X and even PCI-E. There is
>>also a slot connector standard for HT called HTX, but it's not
>>necessarily all that popular.
>
>
> To the best of my knowledge there is only ONE HTX add-in card, an
> Infiniband card from Pathscale. This card was recently used to set
> some world records for low-latency communication in clusters.
>
> The slot is actually VERY similar to PCI-Express (same physical
> connectors) and the specs are designed to make it easy to have both
> PCI-E and HTX on the same board.

I didn't realize that PCI-E and HTX had similar connectors. Is one an
extension of the other (eg. HTX is a few extra slots beyond the PCIE
slots, like VESA was compared to ISA), or something like EISA was to
ISA, with somewhat deeper slots? Or are they totally incompatible but
they look similar?

> Really when you get right down to it, Hypertransport and PCI-Express
> started out with rather different goals but the end result is
> surprisingly similar. I guess there really are only so many ways to
> skin a cat.

Yup.

Yousuf Khan
Anonymous
a b à CPUs
July 26, 2005 4:51:53 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Del Cecchi wrote:
> HT can go maybe a foot, if you are really lucky. Work out the skew
> budgets. At 2000 MT, the board is allocated less than 100 ps as I recall.
>
> PCI-E on the other hand can go several meters.
>
> Totally different approaches.

Which is why PCI-e never got adopted as a CPU to CPU interconnect.

Yousuf Khan
Anonymous
a b à CPUs
July 26, 2005 9:23:10 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

David Kanter wrote:
> One never knows what the future holds. Anyway, it's pretty obvious
> that parallel transmission (read HT) is the way of the past. If you
> look at any high performance interconnect, they are all serial. Talk
> to the Rambus guys, they know what they are doing...

Not quite, HT is a set of multiple serial interfaces. You can go from
one to 16 unidirectional links, one to 16 in the other direction too.
Exactly the same as PCI-e.

> HT was never envisioned to replace PCI-X, PCI or anything else.
> Yousuf, you should at least try to distinguish yourself from AMD PR
> personnel...

It would be much easier if I didn't have to go around correcting
misinformation.

Yousuf Khan
Anonymous
a b à CPUs
July 26, 2005 11:21:31 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

> While a serial (and encoded) link is way easier to handle, the sky is
> not the limit. Consider that at 10Gb/s standard FR-4 board material
> have quite frightening losses, which limits the length you can send
> it. The several meters that Del talk about is on cables, I think.

I don't really know anything about board engineering, but I think that
one thing we might want to consider is that in the near future some CPU
interconnects will simply be on-die; and you can afford to have
ridiculously fast and nice interconnects there.

Also, does PCB have worse loss than cables, as that's what you seem to
be implying.

> And just exactly why would you want to go several meters on a CPU to
> CPU interconnect (at least in the x86 mass-market)?
>
> Sure, the parallel link as other problems, also pointed out by Del,
> but my point here is that blindly claiming that either technology is
> the "right thing" is not a good idea.
>
> Latency, bandwidth, die area, power consumption, and maximum trace
> length should all be considered.

Absolutely, and for a CPU interconnect you probably end up sacrificing
the latter. But as I understand, serial encoding schemes can have a
very small difference in latency from a parallel one (when designed
properly).

> > Now, as to whether serial connections between CPUs is a good idea, I am
> > not entirely sure; I suspect Del is far more qualified to discuss that
> > topic than I am. Generally, serial connections can be driven far
> > faster, but there is slighly longer latency for SERDES.
>
> Definitely - and as we know: money can buy you bandwidth, but latency
> is forever.

Hehehe. We should start selling latency to compete with diamonds,
maybe we can piggyback off of all those De Beers commercials...

> Think of the performace of SDRAMs - while the DDR's have awesome peak
> BW numbers, they rarely translate into real-world benefits that is
> worth taking about.

CPU's always need more bandwidth though, and I think the benefits are
pretty obvious, especially when you start talking about servers with
multiple processors.

David
Anonymous
a b à CPUs
July 27, 2005 12:01:22 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Yousuf Khan wrote:
> David Kanter wrote:
> > One never knows what the future holds. Anyway, it's pretty obvious
> > that parallel transmission (read HT) is the way of the past. If you
> > look at any high performance interconnect, they are all serial. Talk
> > to the Rambus guys, they know what they are doing...
>
> Not quite, HT is a set of multiple serial interfaces. You can go from
> one to 16 unidirectional links, one to 16 in the other direction too.
> Exactly the same as PCI-e.

If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm:

"Serial technologies such as PCI Express and RapidIO require
serial-deserializer interfaces and have the burden of extensive
overhead in encoding parallel data into serial data, embedding clock
information, re-acquiring and decoding the data stream. The parallel
technology of HyperTransport needs no serdes and clock encoding
overhead making it far more efficient in data transfers."

Try to ignore the PR-speak in there, and focus on this part "The
parallel technology of HyperTransport".

HT is bit parallel and delivers at least 2 bits per cycle in parallel;
it's about as parallel as a PCI bus, it just happens to be much more
intelligently designed for the task at hand (and thankfully
unidirectional, and not multidrop).

Now, let me quote someone who knows quite a bit about CPU<->CPU
interconnects:

http://www.realworldtech.com/forums/index.cfm?action=de...

"Using equivelent technology the bit serial scheme will have 2X+ the
datarate per pin. the latency differental is at worst 2 bit times but
can be exactly the same if not better depending on the actual protocol
being used."

PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
is a little worse, but the amount of times it takes to transmit two
bits is pretty darn negligible for double the bandwidth.

David
Anonymous
a b à CPUs
July 27, 2005 1:05:53 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Tue, 26 Jul 2005 12:50:03 -0400, Yousuf Khan <bbbl67@ezrs.com>
wrote:

>Tony Hill wrote:
>> The slot is actually VERY similar to PCI-Express (same physical
>> connectors) and the specs are designed to make it easy to have both
>> PCI-E and HTX on the same board.
>
>I didn't realize that PCI-E and HTX had similar connectors. Is one an
>extension of the other (eg. HTX is a few extra slots beyond the PCIE
>slots, like VESA was compared to ISA), or something like EISA was to
>ISA, with somewhat deeper slots? Or are they totally incompatible but
>they look similar?

More like Slot 1 vs. Slot A. Same physical connector but turned
backwards (or at least that is my understanding of it). The
electrical specs are, not surprisingly, totally incompatible.

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca
Anonymous
a b à CPUs
July 27, 2005 3:24:25 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

"David Kanter" <dkanter@gmail.com> writes:

> Yousuf Khan wrote:
>> Del Cecchi wrote:
>> > HT can go maybe a foot, if you are really lucky. Work out the skew
>> > budgets. At 2000 MT, the board is allocated less than 100 ps as I recall.
>> >
>> > PCI-E on the other hand can go several meters.
>> >
>> > Totally different approaches.
>>
>> Which is why PCI-e never got adopted as a CPU to CPU interconnect.
>
> One never knows what the future holds. Anyway, it's pretty obvious
> that parallel transmission (read HT) is the way of the past. If you
> look at any high performance interconnect, they are all serial. Talk
> to the Rambus guys, they know what they are doing...

While a serial (and encoded) link is way easier to handle, the sky is
not the limit. Consider that at 10Gb/s standard FR-4 board material
have quite frightening losses, which limits the length you can send
it. The several meters that Del talk about is on cables, I think.

And just exactly why would you want to go several meters on a CPU to
CPU interconnect (at least in the x86 mass-market)?

Sure, the parallel link as other problems, also pointed out by Del,
but my point here is that blindly claiming that either technology is
the "right thing" is not a good idea.

Latency, bandwidth, die area, power consumption, and maximum trace
length should all be considered.

> Now, as to whether serial connections between CPUs is a good idea, I am
> not entirely sure; I suspect Del is far more qualified to discuss that
> topic than I am. Generally, serial connections can be driven far
> faster, but there is slighly longer latency for SERDES.

Definitely - and as we know: money can buy you bandwidth, but latency
is forever.

Think of the performace of SDRAMs - while the DDR's have awesome peak
BW numbers, they rarely translate into real-world benefits that is
worth taking about.


Kai
--
Kai Harrekilde-Petersen <khp(at)harrekilde(dot)dk>
Anonymous
a b à CPUs
July 27, 2005 3:24:26 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Kai Harrekilde-Petersen wrote:
> "David Kanter" <dkanter@gmail.com> writes:
>
>
>>Yousuf Khan wrote:
>>
>>>Del Cecchi wrote:
>>>
>>>>HT can go maybe a foot, if you are really lucky. Work out the skew
>>>>budgets. At 2000 MT, the board is allocated less than 100 ps as I recall.
>>>>
>>>>PCI-E on the other hand can go several meters.
>>>>
>>>>Totally different approaches.
>>>
>>>Which is why PCI-e never got adopted as a CPU to CPU interconnect.
>>
>>One never knows what the future holds. Anyway, it's pretty obvious
>>that parallel transmission (read HT) is the way of the past. If you
>>look at any high performance interconnect, they are all serial. Talk
>>to the Rambus guys, they know what they are doing...
>
>
> While a serial (and encoded) link is way easier to handle, the sky is
> not the limit. Consider that at 10Gb/s standard FR-4 board material
> have quite frightening losses, which limits the length you can send
> it. The several meters that Del talk about is on cables, I think.
>
> And just exactly why would you want to go several meters on a CPU to
> CPU interconnect (at least in the x86 mass-market)?
>
> Sure, the parallel link as other problems, also pointed out by Del,
> but my point here is that blindly claiming that either technology is
> the "right thing" is not a good idea.
>
> Latency, bandwidth, die area, power consumption, and maximum trace
> length should all be considered.
>
>
>>Now, as to whether serial connections between CPUs is a good idea, I am
>>not entirely sure; I suspect Del is far more qualified to discuss that
>>topic than I am. Generally, serial connections can be driven far
>>faster, but there is slighly longer latency for SERDES.
>
>
> Definitely - and as we know: money can buy you bandwidth, but latency
> is forever.
>
> Think of the performace of SDRAMs - while the DDR's have awesome peak
> BW numbers, they rarely translate into real-world benefits that is
> worth taking about.
>
>
> Kai
PCI express is an IO expansion network, I don't know where the cpu-cpu
talk came from. Architecturally it is sort of master slave.

As for length, it is limited by the loss budget to 8db of loss at
1.25GHz as I recall, and by ISI to about 100 ps. HT on the other hand
transmits data in parallel with a clock to provide timing and no
alignment so distance is limited primarily by skew as defined in the HT
spec.

The 2.5 Gbit interfaces like PCI-e can go a couple of feet on a
backplane. IB used 20 inches as an objective.

The serdes based standards do indeed have somewhat longer latency due to
the 10 bit penalty on each end. But HT also has to make the data wider
for convenient handling.

This kind of stuff is what I do. So I am familiar with the various
limitations.

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
July 27, 2005 4:57:47 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On 26 Jul 2005 11:05:21 -0700, "David Kanter" <dkanter@gmail.com>
wrote:
....snip previous msgs... - nnn
>One never knows what the future holds. Anyway, it's pretty obvious
>that parallel transmission (read HT) is the way of the past. If you
>look at any high performance interconnect, they are all serial. Talk
>to the Rambus guys, they know what they are doing...
Surely they know - they sue everyone and their mother in law (pun
intended)
nnn
>
>Now, as to whether serial connections between CPUs is a good idea, I am
>not entirely sure; I suspect Del is far more qualified to discuss that
>topic than I am. Generally, serial connections can be driven far
>faster, but there is slighly longer latency for SERDES.
>
>HT was never envisioned to replace PCI-X, PCI or anything else.
>Yousuf, you should at least try to distinguish yourself from AMD PR
>personnel...
>
>David
Anonymous
a b à CPUs
July 27, 2005 7:33:57 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

> >Yousuf Khan wrote:
> >> David Kanter wrote:
> >> > One never knows what the future holds. Anyway, it's pretty obvious
> >> > that parallel transmission (read HT) is the way of the past. If you
> >> > look at any high performance interconnect, they are all serial. Talk
> >> > to the Rambus guys, they know what they are doing...
> >>
> >> Not quite, HT is a set of multiple serial interfaces. You can go from
> >> one to 16 unidirectional links, one to 16 in the other direction too.
> >> Exactly the same as PCI-e.
> >
> >If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm
>
> Knock the colon off to get to this.

Done, sorry about that.

> >"Serial technologies such as PCI Express and RapidIO require
> >serial-deserializer interfaces and have the burden of extensive
> >overhead in encoding parallel data into serial data, embedding clock
> >information, re-acquiring and decoding the data stream. The parallel
> >technology of HyperTransport needs no serdes and clock encoding
> >overhead making it far more efficient in data transfers."
> >
> >Try to ignore the PR-speak in there, and focus on this part "The
> >parallel technology of HyperTransport".
> >
> >HT is bit parallel and delivers at least 2 bits per cycle in parallel;
> >it's about as parallel as a PCI bus, it just happens to be much more
> >intelligently designed for the task at hand (and thankfully
> >unidirectional, and not multidrop).
>
> Selective quoting is never a good idea. Just above, read: "Thus, the
> HyperTransport Technology eliminates the problems associated with high
> speed parallel buses with their many noisy bus signals (multiplexed
> data/address, and clock and control signals) while providing scalable
> bandwidth wherever it is needed in the system."
>
> No, HT is not "about as parallel as a PCI bus".

Actually HT is just as parallel as a PCI bus WRT the bit lanes, which
is all I was speaking about. The reason I didn't bother quoting the
rest is that it doesn't deal with whether the data transmission is
parallel or serial. I'll be the first to admit that HT is alright, but
it could be better if it were serial.

> >Now, let me quote someone who knows quite a bit about CPU<->CPU
> >interconnects:
> >
> >http://www.realworldtech.com/forums/index.cfm?action=de...
> >
> >"Using equivelent technology the bit serial scheme will have 2X+ the
> >datarate per pin. the latency differental is at worst 2 bit times but
> >can be exactly the same if not better depending on the actual protocol
> >being used."
> >
> >PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
> >is a little worse, but the amount of times it takes to transmit two
> >bits is pretty darn negligible for double the bandwidth.
>
> Where do you get double the bandwidth? Both are currently running at
> ~4GB/s at x16. From the same article, next para, as you quoted above:
> "RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
> defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
> specification defines a 2.8 gigatransfers/second data rate".

Perhaps you didn't understand the quote, but it was referring to
abstract, theoretical serial vs. parallel communications, not PCIe vs.
HT. I am not arguing that PCIe has more bandwidth than HT, I am
arguing that bit-serial interconnects are better than bit-arallel ones.
I am further stating (because it is a fact) that HT is bit-parallel.
This limits the speed of HT.

David
Anonymous
a b à CPUs
July 27, 2005 8:45:04 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On 26 Jul 2005 19:21:31 -0700, "David Kanter" <dkanter@gmail.com> wrote:

>> While a serial (and encoded) link is way easier to handle, the sky is
>> not the limit. Consider that at 10Gb/s standard FR-4 board material
>> have quite frightening losses, which limits the length you can send
>> it. The several meters that Del talk about is on cables, I think.
>
>I don't really know anything about board engineering, but I think that
>one thing we might want to consider is that in the near future some CPU
>interconnects will simply be on-die; and you can afford to have
>ridiculously fast and nice interconnects there.
>
>Also, does PCB have worse loss than cables, as that's what you seem to
>be implying.

Cabling? I dunno if this is still cabling but these guys seem to think so:
http://ap.pennnet.com/Articles/Article_Display.cfm?Sect...

--
Rgds, George Macdonald
Anonymous
a b à CPUs
July 27, 2005 8:45:04 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On 26 Jul 2005 20:01:22 -0700, "David Kanter" <dkanter@gmail.com> wrote:

>Yousuf Khan wrote:
>> David Kanter wrote:
>> > One never knows what the future holds. Anyway, it's pretty obvious
>> > that parallel transmission (read HT) is the way of the past. If you
>> > look at any high performance interconnect, they are all serial. Talk
>> > to the Rambus guys, they know what they are doing...
>>
>> Not quite, HT is a set of multiple serial interfaces. You can go from
>> one to 16 unidirectional links, one to 16 in the other direction too.
>> Exactly the same as PCI-e.
>
>If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm:

Knock the colon off to get to this.

>"Serial technologies such as PCI Express and RapidIO require
>serial-deserializer interfaces and have the burden of extensive
>overhead in encoding parallel data into serial data, embedding clock
>information, re-acquiring and decoding the data stream. The parallel
>technology of HyperTransport needs no serdes and clock encoding
>overhead making it far more efficient in data transfers."
>
>Try to ignore the PR-speak in there, and focus on this part "The
>parallel technology of HyperTransport".
>
>HT is bit parallel and delivers at least 2 bits per cycle in parallel;
>it's about as parallel as a PCI bus, it just happens to be much more
>intelligently designed for the task at hand (and thankfully
>unidirectional, and not multidrop).

Selective quoting is never a good idea. Just above, read: "Thus, the
HyperTransport Technology eliminates the problems associated with high
speed parallel buses with their many noisy bus signals (multiplexed
data/address, and clock and control signals) while providing scalable
bandwidth wherever it is needed in the system."

No, HT is not "about as parallel as a PCI bus".

>Now, let me quote someone who knows quite a bit about CPU<->CPU
>interconnects:
>
>http://www.realworldtech.com/forums/index.cfm?action=de...
>
>"Using equivelent technology the bit serial scheme will have 2X+ the
>datarate per pin. the latency differental is at worst 2 bit times but
>can be exactly the same if not better depending on the actual protocol
>being used."
>
>PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
>is a little worse, but the amount of times it takes to transmit two
>bits is pretty darn negligible for double the bandwidth.

Where do you get double the bandwidth? Both are currently running at
~4GB/s at x16. From the same article, next para, as you quoted above:
"RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
specification defines a 2.8 gigatransfers/second data rate".

--
Rgds, George Macdonald
Anonymous
a b à CPUs
July 28, 2005 1:32:36 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

George Macdonald wrote:
> On 26 Jul 2005 20:01:22 -0700, "David Kanter" <dkanter@gmail.com> wrote:
>
>
>>Yousuf Khan wrote:
>>
>>>David Kanter wrote:
>>>
>>>>One never knows what the future holds. Anyway, it's pretty obvious
>>>>that parallel transmission (read HT) is the way of the past. If you
>>>>look at any high performance interconnect, they are all serial. Talk
>>>>to the Rambus guys, they know what they are doing...
>>>
>>>Not quite, HT is a set of multiple serial interfaces. You can go from
>>>one to 16 unidirectional links, one to 16 in the other direction too.
>>>Exactly the same as PCI-e.
>>
>>If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm:
>
>
> Knock the colon off to get to this.
>
>
>>"Serial technologies such as PCI Express and RapidIO require
>>serial-deserializer interfaces and have the burden of extensive
>>overhead in encoding parallel data into serial data, embedding clock
>>information, re-acquiring and decoding the data stream. The parallel
>>technology of HyperTransport needs no serdes and clock encoding
>>overhead making it far more efficient in data transfers."
>>
>>Try to ignore the PR-speak in there, and focus on this part "The
>>parallel technology of HyperTransport".
>>
>>HT is bit parallel and delivers at least 2 bits per cycle in parallel;
>>it's about as parallel as a PCI bus, it just happens to be much more
>>intelligently designed for the task at hand (and thankfully
>>unidirectional, and not multidrop).
>
>
> Selective quoting is never a good idea. Just above, read: "Thus, the
> HyperTransport Technology eliminates the problems associated with high
> speed parallel buses with their many noisy bus signals (multiplexed
> data/address, and clock and control signals) while providing scalable
> bandwidth wherever it is needed in the system."
>
> No, HT is not "about as parallel as a PCI bus".

Knowledge and reading the specifications in question is even a better
idea. HT sends 1 to 32 bits in parallel (the most common
instantiations seem to be 8 or 16 bits of data) accompanied by a clock
for every 8 bits which is used to latch all the bits on the receiving
chip and a framing signal used to mark the start of 4 byte words and
distinguish data from commands. No alignment of the clock and data is
performed so all of the jitter and skew and timing tolerance must be
accomodated by the width of the data bit. The HT spec provides a
detailed allocation of the time in question.
>
>
>>Now, let me quote someone who knows quite a bit about CPU<->CPU
>>interconnects:
>>
>>http://www.realworldtech.com/forums/index.cfm?action=de...
>>
>>"Using equivelent technology the bit serial scheme will have 2X+ the
>>datarate per pin. the latency differental is at worst 2 bit times but
>>can be exactly the same if not better depending on the actual protocol
>>being used."

This isn't true. There is an unavoidable latency penalty associated
with serializing the bytes and deserializing them on the other end.
>>
>>PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
>>is a little worse, but the amount of times it takes to transmit two
>>bits is pretty darn negligible for double the bandwidth.

I would wonder who said that above. What is his name?

The real latency difference comes in error control. If you are going to
wait until the data is known good, you have to wait for 512 bytes in HT,
and to the end of the packet in PCI-E. HT doesn't, so an error causes a
crash. PCI-E and IB do, and retry, so errors are transparent.
>
>
> Where do you get double the bandwidth? Both are currently running at
> ~4GB/s at x16. From the same article, next para, as you quoted above:
> "RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
> defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
> specification defines a 2.8 gigatransfers/second data rate".
>
Go download the spec and tell us how many picoseconds are allowed for
skew and tolerance at the board level.

I would but a dialup is a little slow for that.


--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 28, 2005 6:36:02 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

> >> While a serial (and encoded) link is way easier to handle, the sky is
> >> not the limit. Consider that at 10Gb/s standard FR-4 board material
> >> have quite frightening losses, which limits the length you can send
> >> it. The several meters that Del talk about is on cables, I think.
> >
> > I don't really know anything about board engineering, but I think that
> > one thing we might want to consider is that in the near future some CPU
> > interconnects will simply be on-die; and you can afford to have
> > ridiculously fast and nice interconnects there.
>
> For on-die interconnects, why would you want to go serial at all?

Good point, I don't know what I was thinking about with that line of
thought.

> > Also, does PCB have worse loss than cables, as that's what you seem to
> > be implying.
>
> Standard FR-4 PCB material is quite lousy, so yes, cables have lower
> loss (or rather: tighter margins, which translate into lower loss)
> than FR-4.

Thanks for that info, I didn't realize that. I had always figured it
must have been the other way around.

David
Anonymous
a b à CPUs
July 28, 2005 7:24:04 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

[snip]

> > Selective quoting is never a good idea. Just above, read: "Thus, the
> > HyperTransport Technology eliminates the problems associated with high
> > speed parallel buses with their many noisy bus signals (multiplexed
> > data/address, and clock and control signals) while providing scalable
> > bandwidth wherever it is needed in the system."
> >
> > No, HT is not "about as parallel as a PCI bus".
>
> Knowledge and reading the specifications in question is even a better
> idea. HT sends 1 to 32 bits in parallel (the most common
> instantiations seem to be 8 or 16 bits of data) accompanied by a clock
> for every 8 bits which is used to latch all the bits on the receiving
> chip and a framing signal used to mark the start of 4 byte words and
> distinguish data from commands. No alignment of the clock and data is
> performed so all of the jitter and skew and timing tolerance must be
> accomodated by the width of the data bit. The HT spec provides a
> detailed allocation of the time in question.

> >
> >>Now, let me quote someone who knows quite a bit about CPU<->CPU
> >>interconnects:
> >>
> >>http://www.realworldtech.com/forums/index.cfm?action=de...
> >>
> >>"Using equivelent technology the bit serial scheme will have 2X+ the
> >>datarate per pin. the latency differental is at worst 2 bit times but
> >>can be exactly the same if not better depending on the actual protocol
> >>being used."
>
> This isn't true. There is an unavoidable latency penalty associated
> with serializing the bytes and deserializing them on the other end.

Right, but I believe the point Aaron Spink was making was that a bad
parallel protocol can be worse than a good serial protocol.

> I would wonder who said that above. What is his name?

See above, he's an old friend from comp.arch : ) If you have a
contrary view, you can weigh in back at www.realworldtech.com.

> The real latency difference comes in error control. If you are going to
> wait until the data is known good, you have to wait for 512 bytes in HT,
> and to the end of the packet in PCI-E. HT doesn't, so an error causes a
> crash. PCI-E and IB do, and retry, so errors are transparent.

Interesting. So PCIe has 'better' RAS at the cost of
latency...interesting.

> >
> > Where do you get double the bandwidth? Both are currently running at
> > ~4GB/s at x16. From the same article, next para, as you quoted above:
> > "RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
> > defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
> > specification defines a 2.8 gigatransfers/second data rate".
> >
> Go download the spec and tell us how many picoseconds are allowed for
> skew and tolerance at the board level.
>
> I would but a dialup is a little slow for that.

According to the PDF (page 325):

Uncertainty in CADIN relative to CLKIN due to PCB trace length
mismatch: 10ps

Within pair differential skew of CAD/CTL and CLK due to PCB trace
length mismatch: 5ps

Uncertainty in CADIN relative to other CADIN due to PCB trace length
mismatch: 20ps

PCB interconnect induced jitter caused by reflections, ISI, and
crosstalk CAD/CTL setup side of CLK: 22ps

PCB interconnect induced jitter caused by reflections, ISI, and
crosstalk CAD/CTL hold side of CLK: 22ps

Can you translate that into something useful for me?

David
Anonymous
a b à CPUs
July 28, 2005 9:53:38 PM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

David Kanter wrote:
> [snip]
>
>
>>>Selective quoting is never a good idea. Just above, read: "Thus, the
>>>HyperTransport Technology eliminates the problems associated with high
>>>speed parallel buses with their many noisy bus signals (multiplexed
>>>data/address, and clock and control signals) while providing scalable
>>>bandwidth wherever it is needed in the system."
>>>
>>>No, HT is not "about as parallel as a PCI bus".
>>
>>Knowledge and reading the specifications in question is even a better
>>idea. HT sends 1 to 32 bits in parallel (the most common
>>instantiations seem to be 8 or 16 bits of data) accompanied by a clock
>>for every 8 bits which is used to latch all the bits on the receiving
>>chip and a framing signal used to mark the start of 4 byte words and
>>distinguish data from commands. No alignment of the clock and data is
>>performed so all of the jitter and skew and timing tolerance must be
>>accomodated by the width of the data bit. The HT spec provides a
>>detailed allocation of the time in question.
>
>
>>>>Now, let me quote someone who knows quite a bit about CPU<->CPU
>>>>interconnects:
>>>>
>>>>http://www.realworldtech.com/forums/index.cfm?action=de...
>>>>
>>>>"Using equivelent technology the bit serial scheme will have 2X+ the
>>>>datarate per pin. the latency differental is at worst 2 bit times but
>>>>can be exactly the same if not better depending on the actual protocol
>>>>being used."
>>
>>This isn't true. There is an unavoidable latency penalty associated
>>with serializing the bytes and deserializing them on the other end.
>
>
> Right, but I believe the point Aaron Spink was making was that a bad
> parallel protocol can be worse than a good serial protocol.
>
>
>>I would wonder who said that above. What is his name?
>
>
> See above, he's an old friend from comp.arch : ) If you have a
> contrary view, you can weigh in back at www.realworldtech.com.
>
>
>>The real latency difference comes in error control. If you are going to
>>wait until the data is known good, you have to wait for 512 bytes in HT,
>>and to the end of the packet in PCI-E. HT doesn't, so an error causes a
>>crash. PCI-E and IB do, and retry, so errors are transparent.
>
>
> Interesting. So PCIe has 'better' RAS at the cost of
> latency...interesting.
>
>
>>>Where do you get double the bandwidth? Both are currently running at
>>>~4GB/s at x16. From the same article, next para, as you quoted above:
>>>"RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
>>>defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
>>>specification defines a 2.8 gigatransfers/second data rate".
>>>
>>
>>Go download the spec and tell us how many picoseconds are allowed for
>>skew and tolerance at the board level.
>>
>>I would but a dialup is a little slow for that.
>
>
> According to the PDF (page 325):
>
> Uncertainty in CADIN relative to CLKIN due to PCB trace length
> mismatch: 10ps
>
> Within pair differential skew of CAD/CTL and CLK due to PCB trace
> length mismatch: 5ps
>
> Uncertainty in CADIN relative to other CADIN due to PCB trace length
> mismatch: 20ps
>
> PCB interconnect induced jitter caused by reflections, ISI, and
> crosstalk CAD/CTL setup side of CLK: 22ps
>
> PCB interconnect induced jitter caused by reflections, ISI, and
> crosstalk CAD/CTL hold side of CLK: 22ps
>
> Can you translate that into something useful for me?
>
> David
>
Happy to. PCB delay is about 70 ps/cm. Tracking on pcb delay is 5 to
10 percent. CADIN is a data bit, CLK is the clock. So the traces must
match in length between clk and any of the 9 data/ctl bits by 1.5mm.
Actually, since the tolerance on delay is 5percent, if the length
mismatch was 0, the overall length could only be 3 cm (210 ps) or the 5%
eats your 10 ps.

ISI is the change in transition time due to different pulse widths in
the data, caused by freq dependent loss in interconnect. So if the rise
time is degraded by enough to cause the transitions in 0101010 to be
different place than 00000000011111111100000000 by 22 ps you are also in
violation.

And the within pair skew is the two conductors of the diff pair.
Tolerance gets them too. and 5 ps?

I didn't know Aaron was a PHY type guy. I didn't go to the page due to
dialup and only one phone line.

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 29, 2005 2:08:49 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

"David Kanter" <dkanter@gmail.com> writes:

>> While a serial (and encoded) link is way easier to handle, the sky is
>> not the limit. Consider that at 10Gb/s standard FR-4 board material
>> have quite frightening losses, which limits the length you can send
>> it. The several meters that Del talk about is on cables, I think.
>
> I don't really know anything about board engineering, but I think that
> one thing we might want to consider is that in the near future some CPU
> interconnects will simply be on-die; and you can afford to have
> ridiculously fast and nice interconnects there.

For on-die interconnects, why would you want to go serial at all?

> Also, does PCB have worse loss than cables, as that's what you seem to
> be implying.

Standard FR-4 PCB material is quite lousy, so yes, cables have lower
loss (or rather: tighter margins, which translate into lower loss)
than FR-4.

Regards,


Kai
--
Kai Harrekilde-Petersen <khp(at)harrekilde(dot)dk>
Anonymous
a b à CPUs
July 29, 2005 2:08:50 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

Kai Harrekilde-Petersen wrote:
> "David Kanter" <dkanter@gmail.com> writes:
>
>
>>>While a serial (and encoded) link is way easier to handle, the sky is
>>>not the limit. Consider that at 10Gb/s standard FR-4 board material
>>>have quite frightening losses, which limits the length you can send
>>>it. The several meters that Del talk about is on cables, I think.
>>
>>I don't really know anything about board engineering, but I think that
>>one thing we might want to consider is that in the near future some CPU
>>interconnects will simply be on-die; and you can afford to have
>>ridiculously fast and nice interconnects there.
>
>
> For on-die interconnects, why would you want to go serial at all?
>
>
>>Also, does PCB have worse loss than cables, as that's what you seem to
>>be implying.
>
>
> Standard FR-4 PCB material is quite lousy, so yes, cables have lower
> loss (or rather: tighter margins, which translate into lower loss)
> than FR-4.
>
> Regards,
>
>
> Kai
FR406 isn't bad up to a couple of Gb/s. Above that the dielectric
absorbtion loss tangent gets bad. And cables have larger conductors
than PWB, mostly.

On Die is a whole nother thing because you aren't pin limited.

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”
Anonymous
a b à CPUs
July 29, 2005 5:25:40 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On 27 Jul 2005 15:33:57 -0700, "David Kanter" <dkanter@gmail.com> wrote:

>> >Yousuf Khan wrote:
>> >> David Kanter wrote:
>> >> > One never knows what the future holds. Anyway, it's pretty obvious
>> >> > that parallel transmission (read HT) is the way of the past. If you
>> >> > look at any high performance interconnect, they are all serial. Talk
>> >> > to the Rambus guys, they know what they are doing...
>> >>
>> >> Not quite, HT is a set of multiple serial interfaces. You can go from
>> >> one to 16 unidirectional links, one to 16 in the other direction too.
>> >> Exactly the same as PCI-e.
>> >
>> >If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm
>>
>> Knock the colon off to get to this.
>
>Done, sorry about that.
>
>> >"Serial technologies such as PCI Express and RapidIO require
>> >serial-deserializer interfaces and have the burden of extensive
>> >overhead in encoding parallel data into serial data, embedding clock
>> >information, re-acquiring and decoding the data stream. The parallel
>> >technology of HyperTransport needs no serdes and clock encoding
>> >overhead making it far more efficient in data transfers."
>> >
>> >Try to ignore the PR-speak in there, and focus on this part "The
>> >parallel technology of HyperTransport".
>> >
>> >HT is bit parallel and delivers at least 2 bits per cycle in parallel;
>> >it's about as parallel as a PCI bus, it just happens to be much more
>> >intelligently designed for the task at hand (and thankfully
>> >unidirectional, and not multidrop).
>>
>> Selective quoting is never a good idea. Just above, read: "Thus, the
>> HyperTransport Technology eliminates the problems associated with high
>> speed parallel buses with their many noisy bus signals (multiplexed
>> data/address, and clock and control signals) while providing scalable
>> bandwidth wherever it is needed in the system."
>>
>> No, HT is not "about as parallel as a PCI bus".
>
>Actually HT is just as parallel as a PCI bus WRT the bit lanes, which
>is all I was speaking about. The reason I didn't bother quoting the
>rest is that it doesn't deal with whether the data transmission is
>parallel or serial. I'll be the first to admit that HT is alright, but
>it could be better if it were serial.

You have to define better - add serialized and lose on latency. I see HT
as more of a hybrid inter-connect, with its packetization and lack of
control signals or multiplexing. It's somewhat similar to the P4 FSB with
clock signals allocated to 8-bit widths instead of 16-bit. I believe some
confusion has arisen here since I'm almost sure that in its original form,
LDT was said to be a serial interconnect.

>> >Now, let me quote someone who knows quite a bit about CPU<->CPU
>> >interconnects:
>> >
>> >http://www.realworldtech.com/forums/index.cfm?action=de...
>> >
>> >"Using equivelent technology the bit serial scheme will have 2X+ the
>> >datarate per pin. the latency differental is at worst 2 bit times but
>> >can be exactly the same if not better depending on the actual protocol
>> >being used."
>> >
>> >PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
>> >is a little worse, but the amount of times it takes to transmit two
>> >bits is pretty darn negligible for double the bandwidth.
>>
>> Where do you get double the bandwidth? Both are currently running at
>> ~4GB/s at x16. From the same article, next para, as you quoted above:
>> "RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
>> defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
>> specification defines a 2.8 gigatransfers/second data rate".
>
>Perhaps you didn't understand the quote, but it was referring to
>abstract, theoretical serial vs. parallel communications, not PCIe vs.
>HT. I am not arguing that PCIe has more bandwidth than HT, I am
>arguing that bit-serial interconnects are better than bit-arallel ones.
> I am further stating (because it is a fact) that HT is bit-parallel.
>This limits the speed of HT.

Horses for courses I guess - HT has to do the job of CPU interconnect which
includes non-local memory accesses. Would it really be better to take the
latency hit on those?

I'm afraid your para above starting "PCIe is bit serial, HT, as I...." on
re-read still indicates to me that you were suggesting that PCIe has
"double the bandwidth" of HT.

--
Rgds, George Macdonald
Anonymous
a b à CPUs
July 29, 2005 5:25:41 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

On Wed, 27 Jul 2005 21:32:36 -0500, Del Cecchi <cecchinospam@us.ibm.com>
wrote:

>George Macdonald wrote:
>> On 26 Jul 2005 20:01:22 -0700, "David Kanter" <dkanter@gmail.com> wrote:
>>
>>
>>>Yousuf Khan wrote:
>>>
>>>>David Kanter wrote:
>>>>
>>>>>One never knows what the future holds. Anyway, it's pretty obvious
>>>>>that parallel transmission (read HT) is the way of the past. If you
>>>>>look at any high performance interconnect, they are all serial. Talk
>>>>>to the Rambus guys, they know what they are doing...
>>>>
>>>>Not quite, HT is a set of multiple serial interfaces. You can go from
>>>>one to 16 unidirectional links, one to 16 in the other direction too.
>>>>Exactly the same as PCI-e.
>>>
>>>If I may quote from http://www.hypertransport.org/tech/tech_faqs.cfm:
>>
>>
>> Knock the colon off to get to this.
>>
>>
>>>"Serial technologies such as PCI Express and RapidIO require
>>>serial-deserializer interfaces and have the burden of extensive
>>>overhead in encoding parallel data into serial data, embedding clock
>>>information, re-acquiring and decoding the data stream. The parallel
>>>technology of HyperTransport needs no serdes and clock encoding
>>>overhead making it far more efficient in data transfers."
>>>
>>>Try to ignore the PR-speak in there, and focus on this part "The
>>>parallel technology of HyperTransport".
>>>
>>>HT is bit parallel and delivers at least 2 bits per cycle in parallel;
>>>it's about as parallel as a PCI bus, it just happens to be much more
>>>intelligently designed for the task at hand (and thankfully
>>>unidirectional, and not multidrop).
>>
>>
>> Selective quoting is never a good idea. Just above, read: "Thus, the
>> HyperTransport Technology eliminates the problems associated with high
>> speed parallel buses with their many noisy bus signals (multiplexed
>> data/address, and clock and control signals) while providing scalable
>> bandwidth wherever it is needed in the system."
>>
>> No, HT is not "about as parallel as a PCI bus".
>
>Knowledge and reading the specifications in question is even a better
>idea. HT sends 1 to 32 bits in parallel (the most common
>instantiations seem to be 8 or 16 bits of data) accompanied by a clock
>for every 8 bits which is used to latch all the bits on the receiving
>chip and a framing signal used to mark the start of 4 byte words and
>distinguish data from commands. No alignment of the clock and data is
>performed so all of the jitter and skew and timing tolerance must be
>accomodated by the width of the data bit. The HT spec provides a
>detailed allocation of the time in question.

Yeah the spec is better but looooong and I'm lazy and not able to see PCIe
specs anyway, since they cost $$.:-)

>>>Now, let me quote someone who knows quite a bit about CPU<->CPU
>>>interconnects:
>>>
>>>http://www.realworldtech.com/forums/index.cfm?action=de...
>>>
>>>"Using equivelent technology the bit serial scheme will have 2X+ the
>>>datarate per pin. the latency differental is at worst 2 bit times but
>>>can be exactly the same if not better depending on the actual protocol
>>>being used."
>
>This isn't true. There is an unavoidable latency penalty associated
>with serializing the bytes and deserializing them on the other end.
>>>
>>>PCIe is bit serial, HT, as I explained above, is not. Yes, the latency
>>>is a little worse, but the amount of times it takes to transmit two
>>>bits is pretty darn negligible for double the bandwidth.
>
>I would wonder who said that above. What is his name?

If you mean the quote from RealWorldTech it was Aaron Spink.

>The real latency difference comes in error control. If you are going to
>wait until the data is known good, you have to wait for 512 bytes in HT,
>and to the end of the packet in PCI-E. HT doesn't, so an error causes a
>crash. PCI-E and IB do, and retry, so errors are transparent.

Are you saying that HT implementations just ignore the CRC, or it doesn't
work right at the 64-bits into the next window?

>> Where do you get double the bandwidth? Both are currently running at
>> ~4GB/s at x16. From the same article, next para, as you quoted above:
>> "RapidIO defines a data rate of 3.125 gigabit/second, while PCI Express
>> defines a 2.5 gigabit/second data rate. The latest 2.0 HyperTransport
>> specification defines a 2.8 gigatransfers/second data rate".
>>
>Go download the spec and tell us how many picoseconds are allowed for
>skew and tolerance at the board level.

From a Rev2.00 I DL'd last December there are several numbers listed for
relative skews but the tightest is for "Within pair differential skew of
CAD/CTL and CLK due to PCB trace length mismatch" the value listed for a
1GHz clock, which is common currently, it shows 5ps.

--
Rgds, George Macdonald
Anonymous
a b à CPUs
July 30, 2005 1:17:31 AM

Archived from groups: comp.sys.ibm.pc.hardware.chips (More info?)

> You have to define better - add serialized and lose on latency.

Aaron's point was that the loss on latency isn't that bad if you do
things right.

> I see HT
> as more of a hybrid inter-connect, with its packetization and lack of
> control signals or multiplexing. It's somewhat similar to the P4 FSB with
> clock signals allocated to 8-bit widths instead of 16-bit. I believe some
> confusion has arisen here since I'm almost sure that in its original form,
> LDT was said to be a serial interconnect.

> >Perhaps you didn't understand the quote, but it was referring to
> >abstract, theoretical serial vs. parallel communications, not PCIe vs.
> >HT. I am not arguing that PCIe has more bandwidth than HT, I am
> >arguing that bit-serial interconnects are better than bit-arallel ones.
> > I am further stating (because it is a fact) that HT is bit-parallel.
> >This limits the speed of HT.
>
> Horses for courses I guess - HT has to do the job of CPU interconnect which
> includes non-local memory accesses. Would it really be better to take the
> latency hit on those?

If the latency hit is small, quite possibly.

> I'm afraid your para above starting "PCIe is bit serial, HT, as I...." on
> re-read still indicates to me that you were suggesting that PCIe has
> "double the bandwidth" of HT.

Well, that wasn't the case.

David
!