"Don't wanna be a pain but wouldn't you hit some limitions in data transfer rates."
Not a pain at all, valid observation. Truth is, I'm not sure what their limitations are, I've only ever worked with the 5 PCI setup...
"Besides the pci slots would have to go separate and in some weird shaped box. not to mention the cables. And if you've got that kind of money buy more pc and a switch!"
That's the way I'd do it. I have a 6 node dual processor P-Pro 200(I know, breathtaking) cluster with a 100mbit backplane, which I've used to run weather sims(slowly) and to learn more about paralell programming. So i'm with you there(although my new dual opteron system blows it out of the water).
The thing is, I suspect you'll get better performance through a PCI bus than through a gigabit backplane. Last I checked, in megabytes(as opposed to bits), gigabits bandwidth was 114mb/sec. The earliest version of the PCI bus was 132mb/sec, and the later versions go up to 1066mb/sec. But assuming you could only get 132mb/sec bandwidth, in addition to the slightly larger bandwidth, you should still experience far less latency than you will by going from the processor, to the PCI bus, to your NIC, out your NIC to the switch, then to the master nodes NIC, to the bus, to the processor and back again through the same chain. Keep in mind also that this latency is compounded by the fact that when mulitple nodes are communicating, their communication with the master is impacted by the latency experienced by other nodes.
In the PCI setup, you go from the cpu or gpu on the card, to the PCI bus, to the processor and back again. That can make a huge difference in big jobs, or anything you're trying to to do "real time".
Of course, this is all theory on my part as I know zip, zilch, and squat about how their tech works and just a smidgeon about how beowulf clustering works. I do know I've learned the latency lesson well in configuring my own cluster and I think my theory is sound in thinking that the PCI setup would have substantially less, and it probably has closer to a gbyte of bandwidth.
<i>SCO is to Linux what a flea is to a dog.</i>