Sign in with
Sign up | Sign in
Your question

four 9800GX2 cards: will it work?

Last response: in Graphics & Displays
Share
March 19, 2008 3:28:30 PM

I work as a Computer Scientist at an academic image processing department. In my research, I regularly work with nvidia Cuda (http://www.nvidia.com/object/cuda_home.html) for high performance computing on the graphics card (GPGPU). These are general computations that have nothing to do with rendering 3D scenes. For GPGPU, SLI is not required. Therefore, it seems possible to me to put four 9800GX2 cards on a single motherboard, for example the MSI K9A2 Platinum V2 (http://global.msi.com.tw/index.php?func=proddesc&prod_no=1395&maincat_no=1) that has four (physical) PCI-Express X16 slots with double spacing between the slots.

When using CUDA, SLI is not used and each 9800GX2 is "seen" by the system as two separate GPUs. Therefore I am hoping to get access to eight separate GPUs by using four 9800GX2 cards. My computations can be distributed over these eight GPUs without requiring communication between the different cards (or between the two GPUs on each card).

So far, I have not found any information from others who have tried this before. Cooling and power are obvious problems that can probably be dealt with. Any ideas on problems that I can expect on the software side? For example, will the (non-SLI) driver see the four 9800GX2 cards as eight independent GPUs? Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?

Any comments would be appreciated!

More about : 9800gx2 cards work

March 19, 2008 3:32:27 PM

While this has nothing to do with anything I would ever do, I am extremely curious about this.
March 19, 2008 4:40:48 PM

Could you by any chance install 3DMark06 and Crysis on that machine.
Related resources
March 19, 2008 5:16:48 PM

Interesting questions - I cannot help here and I'm not sure this is the right forum to ask (see previous posts :-/) , try talkig to NVidia
Questions this forum might be able to help with are:

Do the non-SLI drivers 'see' a SINGLE 9800GX as two independent GPU's?

and

Anyone out there know of a MB that will actually TAKE four 16x PCIe cards?
March 19, 2008 5:22:09 PM

dev1se said:
Could you by any chance install 3DMark06 and Crysis on that machine.


Yeah, I am curious about how the performance would be. But, since he isn't going to use SLI, it won't really be any different than having a single card, unless he ran the game across multiple monitors.
March 19, 2008 5:24:11 PM

I believe the system will see all 8 GPUs, it will see 2 for every card put in. I have not tested this but I am a computer repair technician and work with computer hardware a lot. From my knowledge of how they work it will see all 8 GPUs and you can assign the driver for each of them. They will not function in unison because of no SLI support but they will be operable. So what you plan to do should, in theory, work just fine.

Oh, and yes i would reccommend a 790FX board like the MSI one you posted. It doesnt matter that it is AMD and only does CrossfireX because you are not looking for Crossfire or SLI
March 19, 2008 5:33:24 PM

I would ask nVidia if I were you, they are the only ones that will know all the factors involved.

It should be interesting to see if any other bottlenecks appear in this setup.
March 19, 2008 10:06:23 PM

holy crap. I have no idea, It would be sweet if it did though. Can I have your computer after you're done researching with it?
March 19, 2008 10:39:44 PM

Thanks a lot for all your inputs! I have in fact contacted NVidia about this, but as I did not yet receive a reply and I am quite anxious to start ordering components :D , I decided to post the question here too.

I work in Belgium, where the most powerful supercomputer is capable of 2TFlops. If this works, the single box should have a peak performance beating that :lol:  (NVidia claims ~500 GFlops for a single 8800GTX). I know this is just true for very specific applications, but still it would be quite remarkable.

About running 3DMark: the 790FX board is not capable of SLI, so it doesn't make much sense to benchmark the system on 3D capabilities. Has anyone ever tried the MSI K9A2 Platinum board with four graphics cards at all (for example, four ATI cards in cross-fire setup)?
March 19, 2008 10:44:33 PM

You will have one limitation IF it works. The 9800GTX shares 1 PCI slot, even on a PCI-2 board, it will be limited to PCI-1 speeds, as well as 8x per card/slot
a b Î Nvidia
March 19, 2008 11:04:40 PM

There's no reason why M$ shouldn't see all 8 VPU, the only issue is whether or not nVidia has opened up Cuda to recognize them.

Support for multi-VPU GPGPU work far preceeded their gaming counterparts with both IHV's getting them to work prior to any gaming support.

If nV doesn't get you a quick answer then get in touch with some of the large software devs like Rapidmind or Acceleware, or even the competition PeakStream.

Not sure if the Tesla experiences would apply as eaily because of it's more focused purpose, but they are into multi-rack rigs there.

Probably a good place to ask would be the GPGPU.org forums, I check there every once in a while for F@H updates.
March 20, 2008 6:18:35 AM

kjoost, I see a promblem with your setup. As I remember CUDA needs at least one processor core per GPU core, at least to work effectively. The MSI motherboard only takes one quad core CPU, so you shoukd consider something Intel Skulltrail or some other dual Xeon motherboards.
March 20, 2008 7:18:16 AM

Hi guys, thanks again!

jaydeejohn: You are absolutely right about the bandwidth. For my applications (tomographic image reconstruction), bandwidth from CPU to GPU is not the bottleneck, but in general this will be a problem.

ahu1: I am aware of that problem, but I was under the impression that this was only an issue for early versions of CUDA, for which the CPU thread blocked on execution of a kernel on the GPU. In any case, I will check it out first with nvidia.

March 21, 2008 9:34:36 AM

In the mean while, I got some more info from the NVIDIA forums: see http://forums.nvidia.com/index.php?showtopic=62618

The one-core-per-GPU requirement is not a hard constraint, more like advice from NVIDIA.

It seems that noone can guarantee that it won't work, but also noone has ever tried. MSI say that their board should work, "in theory", with 8 GPUs, but they don't want to give any guarantees either...

Anyway, I have managed to get my boss behind this effort and we will now start ordering some components. I will post pictures as soon as anything is running (it may take a while, though) :) 
March 21, 2008 10:51:59 AM

kjoost said:
In the mean while, I got some more info from the NVIDIA forums: see http://forums.nvidia.com/index.php?showtopic=62618

The one-core-per-GPU requirement is not a hard constraint, more like advice from NVIDIA.

It seems that noone can guarantee that it won't work, but also noone has ever tried. MSI say that their board should work, "in theory", with 8 GPUs, but they don't want to give any guarantees either...

Anyway, I have managed to get my boss behind this effort and we will now start ordering some components. I will post pictures as soon as anything is running (it may take a while, though) :) 


heh i wouldnt trust MSI for anything important

intel skulltrail will give you reliability and full pcie bandwidth, and more memory bandwidth for that matter - the better way to go.
March 21, 2008 11:58:29 AM

I agree that Skulltrail would be a better choice, but... Do you know of any Skulltrail motherboard that has double slot spacing between all four PCI-Express X16 slots?
March 21, 2008 5:52:55 PM

I guess there's only Skulltrail motherboard and one of the PCIe slots is single spaced. But there are also some dual-socket Opteron motherboards eg. Tyan with four PCIe x16 slots, so check other options also.

About the one-core-per-GPU requirement: AFAIK there are some latency issues with shorter CUDA kernels if you don't have enough CPU cores.

Anyway, keep us informed... Interesting project you have there :) 
March 30, 2008 2:01:45 AM

I am very interested to see what happens, please keep us updated; this forum is in my bookmarks : )
March 31, 2008 9:45:03 PM

Just a quick update: I have ordered several components, including the case (Lian Li PC-P80) and PSU (Thermaltake Toughpower 1500W). The ETA for the PSU is yet unclear, so I hope it will work. I will keep you posted :) 
April 3, 2008 1:58:35 AM

Could you do four GTX's in SLI?
a b Î Nvidia
April 3, 2008 6:56:22 AM

No benefit to that over four GX2s.

4 GPUs compared to the 8 he'll have with the four GX2s.

Only reason to go that route would be if their computer power were twice as much or more.
April 17, 2008 9:49:05 AM

Hi guys,

Just a quick update, as I am currently travelling: the beast has been built!
It appears to work (I mean boot, run standard benchmarks), although we have not yet tested its supercomputing abilities yet.

Expect pictures somewhere next week :) 
a b Î Nvidia
April 17, 2008 3:29:11 PM

Now get another 3 and make a nice little cluster to take on Sandia Labs. :D 
April 24, 2008 6:52:32 PM

I'm sorry to keep you waiting, but there will be some delay (probably about 2-3 more weeks) due to other urgent work issues. I will keep you posted. Just to show that the whole thing is for real, here

is a picture of the components used (although one graphics card is missing on the picture) :) 
May 14, 2008 5:59:17 AM

Hi kjoost

there are somthing about CUDA and more than one Graphic Card
http://forums.nvidia.com/index.php?showtopic=30740



I'm using CUDA for GPGPU researching now. I'm curious about how you test the scientific performance of the platform.
We are trying to rewrite the 464.H264 from SPEC CPU2006 to test CUDA's performance
Hope we can discuss it deeply.
pls contact: yuxing.tang [ at ] gmail.com
May 17, 2008 10:34:27 PM

Am I the only one wondering if 1500Watt PSU is power enough at all?

Under max. load this card hurles 360Watts per card!

I imagine you're going to have power issues, when the cards start sucking power under stress load!
a b Î Nvidia
May 18, 2008 10:22:44 PM

Eizo said:
Am I the only one wondering if 1500Watt PSU is power enough at all?

Under max. load this card hurles 360Watts per card!


Where are you getting that number from? It doesn't even have the connectors to provide that much power without relying on a PCIe 2.0 connection as a requirement not an option.
a b Î Nvidia
May 19, 2008 9:34:06 AM

Hang on a minuite is it me or should this bit be raising the alarm bells.
"Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?"
XP only supports 2 GPU's dosent it ?
Mactronix
May 19, 2008 10:04:55 AM

HUGELY interesting Post! Awesome setup man :)  Looking fwd to hear from u further!
May 19, 2008 4:37:18 PM

Bump for greater justice...and news ofc !!
a b Î Nvidia
May 19, 2008 7:41:12 PM

mactronix said:

XP only supports 2 GPU's dosent it ?


No, and you're thinking about the render-ahead limitation in DirectX (not openGL), which doesn't apply to GPGPU apps the OP is using.
a b Î Nvidia
May 19, 2008 8:23:09 PM

TheGreatGrapeApe said:
No, and you're thinking about the render-ahead limitation in DirectX (not openGL), which doesn't apply to GPGPU apps the OP is using.

Slightly off topic.
Ok i know you well enough to take your word for it Ape. :) 
Its just i have seen plenty of forum posts asking about tri sli on xp and everyone on 3-4 differant forums have said words to this effect.
"Tri-SLI will not work in XP. XP is only capable of using 2 GPU's at a time. No matter what drivers are used. Even drivers coded by Jesus himself can't get past the 2 GPU limitation in XP. Sorry =o(, a Vista install is soon peeking over your horizon!"
Can you send me a link or some such when you get time.
Thanks
Mactronix :) 
Ok guys and girls i will butt out now as i obviously am out of my depth here.
May 19, 2008 8:32:00 PM

Is this thread a joke?
And 4 9800GX2?
Are you boasting?
Looks funny to me!
May 20, 2008 1:34:51 AM

kjoost said:
I work as a Computer Scientist at an academic image processing department. In my research, I regularly work with nvidia Cuda (http://www.nvidia.com/object/cuda_home.html) for high performance computing on the graphics card (GPGPU). These are general computations that have nothing to do with rendering 3D scenes. For GPGPU, SLI is not required. Therefore, it seems possible to me to put four 9800GX2 cards on a single motherboard, for example the MSI K9A2 Platinum V2 (http://global.msi.com.tw/index.php?func=proddesc&prod_no=1395&maincat_no=1) that has four (physical) PCI-Express X16 slots with double spacing between the slots.

When using CUDA, SLI is not used and each 9800GX2 is "seen" by the system as two separate GPUs. Therefore I am hoping to get access to eight separate GPUs by using four 9800GX2 cards. My computations can be distributed over these eight GPUs without requiring communication between the different cards (or between the two GPUs on each card).

So far, I have not found any information from others who have tried this before. Cooling and power are obvious problems that can probably be dealt with. Any ideas on problems that I can expect on the software side? For example, will the (non-SLI) driver see the four 9800GX2 cards as eight independent GPUs? Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?

Any comments would be appreciated!


I think CUDA works with 4 pieces of 9800GX2 (8 GPU) without problem which is just highly depended on how you can modify the original driver by C language, however I just wonder if you need 8 cores CPU also in order to avoid bottleneck of computing. Except the monstor board D5400XS from Intel can support 2 CPU and 4x PCI-E 16 slots can meet this target. The only problem is no games could support CUDA coz CUDA main purpose is for performance computing not for performance 3D games.
a b Î Nvidia
May 20, 2008 3:55:18 PM

mactronix said:
Slightly off topic.
Ok i know you well enough to take your word for it Ape. :) 
Its just i have seen plenty of forum posts asking about tri sli on xp and everyone on 3-4 differant forums have said words to this effect.
"Tri-SLI will not work in XP. XP is only capable of using 2 GPU's at a time. No matter what drivers are used. Even drivers coded by Jesus himself can't get past the 2 GPU limitation in XP. Sorry =o(, a Vista install is soon peeking over your horizon!"
Can you send me a link or some such when you get time.


Yeah, this is a limit for AFR under XP and DirectX.
Here's a quick little blurb on it from the TechReport;
http://techreport.com/articles.x/14284/2

And the limit is for AFR so your could still do 3 frames ahead and effectively use 3 VPUs for just AFR, plus you could still have up to 6 VPUs rendering in AFR+SFR so A1+A2/B1+B2/C1+C2/A1+A2/.....
My understanding is that this limit does not apply to OpenGL, as mentioned in this HardOCP article;
http://enthusiast.hardocp.com/article.html?art=MTExNiwy...

If I can find more later, I'll re-post.

However for what the OP is using them for, this is not an issue since it's not using them for rendering frames/frames ahead, but using them as co-processors, and basically he's just adding Gflops to his processing power.


a b Î Nvidia
May 21, 2008 6:55:35 AM


Thanks for that will read them later any more info apreciated.
Mactronix
May 26, 2008 8:54:28 PM

Hi guys, it took a long time, but I will finally be able to report back to you on the result of this thread. Next Wednesday (May 28th), we will launch a website on our GPU system, with lots of information on the background of our application, system specs, benchmarks, etc.

Stay tuned!
May 28, 2008 1:10:41 PM

Finally, I can answer the fundamental question of this thread:

FOUR 9800GX2 cards: will it work?

Check out our website http://fastra.ua.ac.be for the answer... ;) 


Thanks for all your help; we couldn't have done it without you :) 
May 28, 2008 1:35:04 PM

Good Luck dude, all the best!

PS: Are you Batenburg?
May 28, 2008 1:37:06 PM

Your are! Dr. K. Joost Batenburg!
May 28, 2008 2:00:33 PM

Good luck with your ptroduct, may it save many lives....and make you a few bucks and honor as well
May 28, 2008 2:04:26 PM

Yeah! I visited the site and the results are motherfragingly impressive!
This may be going to be some really great breakthrough.
Congratulations!
May 28, 2008 3:20:48 PM

romulus47plus1 said:
Good Luck dude, all the best!

PS: Are you Batenburg?


Yes, that's me. :) 
a b Î Nvidia
May 28, 2008 3:51:55 PM

DANG It !! [:zorg:2]

I thought I was using my talents for Evil not Good; something like building a Supercomputer for a war mongering Banana Republic!! MMmmmm Bananas........

Man I'm gonna get razzed about this at the next meeting of evil doers next Thursday. :sweat: 


__
PS, sorry about the edit, undid it I think, mis-clicked the button. Or just being Evil. ;) 
May 28, 2008 3:57:40 PM

Maybe if your photo/icon had a cape? heheh
!