four 9800GX2 cards: will it work?

I work as a Computer Scientist at an academic image processing department. In my research, I regularly work with nvidia Cuda (http://www.nvidia.com/object/cuda_home.html) for high performance computing on the graphics card (GPGPU). These are general computations that have nothing to do with rendering 3D scenes. For GPGPU, SLI is not required. Therefore, it seems possible to me to put four 9800GX2 cards on a single motherboard, for example the MSI K9A2 Platinum V2 (http://global.msi.com.tw/index.php?func=proddesc&prod_no=1395&maincat_no=1) that has four (physical) PCI-Express X16 slots with double spacing between the slots.

When using CUDA, SLI is not used and each 9800GX2 is "seen" by the system as two separate GPUs. Therefore I am hoping to get access to eight separate GPUs by using four 9800GX2 cards. My computations can be distributed over these eight GPUs without requiring communication between the different cards (or between the two GPUs on each card).

So far, I have not found any information from others who have tried this before. Cooling and power are obvious problems that can probably be dealt with. Any ideas on problems that I can expect on the software side? For example, will the (non-SLI) driver see the four 9800GX2 cards as eight independent GPUs? Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?

Any comments would be appreciated!
46 answers Last reply
More about four 9800gx2 cards work
  1. While this has nothing to do with anything I would ever do, I am extremely curious about this.
  2. Could you by any chance install 3DMark06 and Crysis on that machine.
  3. Interesting questions - I cannot help here and I'm not sure this is the right forum to ask (see previous posts :-/) , try talkig to NVidia
    Questions this forum might be able to help with are:

    Do the non-SLI drivers 'see' a SINGLE 9800GX as two independent GPU's?

    and

    Anyone out there know of a MB that will actually TAKE four 16x PCIe cards?
  4. dev1se said:
    Could you by any chance install 3DMark06 and Crysis on that machine.


    Yeah, I am curious about how the performance would be. But, since he isn't going to use SLI, it won't really be any different than having a single card, unless he ran the game across multiple monitors.
  5. I believe the system will see all 8 GPUs, it will see 2 for every card put in. I have not tested this but I am a computer repair technician and work with computer hardware a lot. From my knowledge of how they work it will see all 8 GPUs and you can assign the driver for each of them. They will not function in unison because of no SLI support but they will be operable. So what you plan to do should, in theory, work just fine.

    Oh, and yes i would reccommend a 790FX board like the MSI one you posted. It doesnt matter that it is AMD and only does CrossfireX because you are not looking for Crossfire or SLI
  6. I would ask nVidia if I were you, they are the only ones that will know all the factors involved.

    It should be interesting to see if any other bottlenecks appear in this setup.
  7. holy crap. I have no idea, It would be sweet if it did though. Can I have your computer after you're done researching with it?
  8. Thanks a lot for all your inputs! I have in fact contacted NVidia about this, but as I did not yet receive a reply and I am quite anxious to start ordering components :D, I decided to post the question here too.

    I work in Belgium, where the most powerful supercomputer is capable of 2TFlops. If this works, the single box should have a peak performance beating that :lol: (NVidia claims ~500 GFlops for a single 8800GTX). I know this is just true for very specific applications, but still it would be quite remarkable.

    About running 3DMark: the 790FX board is not capable of SLI, so it doesn't make much sense to benchmark the system on 3D capabilities. Has anyone ever tried the MSI K9A2 Platinum board with four graphics cards at all (for example, four ATI cards in cross-fire setup)?
  9. You will have one limitation IF it works. The 9800GTX shares 1 PCI slot, even on a PCI-2 board, it will be limited to PCI-1 speeds, as well as 8x per card/slot
  10. There's no reason why M$ shouldn't see all 8 VPU, the only issue is whether or not nVidia has opened up Cuda to recognize them.

    Support for multi-VPU GPGPU work far preceeded their gaming counterparts with both IHV's getting them to work prior to any gaming support.

    If nV doesn't get you a quick answer then get in touch with some of the large software devs like Rapidmind or Acceleware, or even the competition PeakStream.

    Not sure if the Tesla experiences would apply as eaily because of it's more focused purpose, but they are into multi-rack rigs there.

    Probably a good place to ask would be the GPGPU.org forums, I check there every once in a while for F@H updates.
  11. kjoost, I see a promblem with your setup. As I remember CUDA needs at least one processor core per GPU core, at least to work effectively. The MSI motherboard only takes one quad core CPU, so you shoukd consider something Intel Skulltrail or some other dual Xeon motherboards.
  12. Hi guys, thanks again!

    jaydeejohn: You are absolutely right about the bandwidth. For my applications (tomographic image reconstruction), bandwidth from CPU to GPU is not the bottleneck, but in general this will be a problem.

    ahu1: I am aware of that problem, but I was under the impression that this was only an issue for early versions of CUDA, for which the CPU thread blocked on execution of a kernel on the GPU. In any case, I will check it out first with nvidia.
  13. In the mean while, I got some more info from the NVIDIA forums: see http://forums.nvidia.com/index.php?showtopic=62618

    The one-core-per-GPU requirement is not a hard constraint, more like advice from NVIDIA.

    It seems that noone can guarantee that it won't work, but also noone has ever tried. MSI say that their board should work, "in theory", with 8 GPUs, but they don't want to give any guarantees either...

    Anyway, I have managed to get my boss behind this effort and we will now start ordering some components. I will post pictures as soon as anything is running (it may take a while, though) :)
  14. kjoost said:
    In the mean while, I got some more info from the NVIDIA forums: see http://forums.nvidia.com/index.php?showtopic=62618

    The one-core-per-GPU requirement is not a hard constraint, more like advice from NVIDIA.

    It seems that noone can guarantee that it won't work, but also noone has ever tried. MSI say that their board should work, "in theory", with 8 GPUs, but they don't want to give any guarantees either...

    Anyway, I have managed to get my boss behind this effort and we will now start ordering some components. I will post pictures as soon as anything is running (it may take a while, though) :)


    heh i wouldnt trust MSI for anything important

    intel skulltrail will give you reliability and full pcie bandwidth, and more memory bandwidth for that matter - the better way to go.
  15. I agree that Skulltrail would be a better choice, but... Do you know of any Skulltrail motherboard that has double slot spacing between all four PCI-Express X16 slots?
  16. I guess there's only Skulltrail motherboard and one of the PCIe slots is single spaced. But there are also some dual-socket Opteron motherboards eg. Tyan with four PCIe x16 slots, so check other options also.

    About the one-core-per-GPU requirement: AFAIK there are some latency issues with shorter CUDA kernels if you don't have enough CPU cores.

    Anyway, keep us informed... Interesting project you have there :)
  17. I am very interested to see what happens, please keep us updated; this forum is in my bookmarks : )
  18. Just a quick update: I have ordered several components, including the case (Lian Li PC-P80) and PSU (Thermaltake Toughpower 1500W). The ETA for the PSU is yet unclear, so I hope it will work. I will keep you posted :)
  19. Good Luck! :sol:
  20. Could you do four GTX's in SLI?
  21. No benefit to that over four GX2s.

    4 GPUs compared to the 8 he'll have with the four GX2s.

    Only reason to go that route would be if their computer power were twice as much or more.
  22. Hi guys,

    Just a quick update, as I am currently travelling: the beast has been built!
    It appears to work (I mean boot, run standard benchmarks), although we have not yet tested its supercomputing abilities yet.

    Expect pictures somewhere next week :)
  23. Now get another 3 and make a nice little cluster to take on Sandia Labs. :D
  24. I'm sorry to keep you waiting, but there will be some delay (probably about 2-3 more weeks) due to other urgent work issues. I will keep you posted. Just to show that the whole thing is for real, here

    is a picture of the components used (although one graphics card is missing on the picture) :)
  25. Hi kjoost

    there are somthing about CUDA and more than one Graphic Card
    http://forums.nvidia.com/index.php?showtopic=30740


    I'm using CUDA for GPGPU researching now. I'm curious about how you test the scientific performance of the platform.
    We are trying to rewrite the 464.H264 from SPEC CPU2006 to test CUDA's performance
    Hope we can discuss it deeply.
    pls contact: yuxing.tang [ at ] gmail.com
  26. Am I the only one wondering if 1500Watt PSU is power enough at all?

    Under max. load this card hurles 360Watts per card!

    I imagine you're going to have power issues, when the cards start sucking power under stress load!
  27. If he does run into watt/amp probs which i doubt he will he could always throw 2x1500w in this case. http://www.newegg.com/Product/Product.aspx?Item=N82E16811129028

    EDIT: I just realized that 4GPUS wouldn't fit in this case sorry for the inconvenience
  28. Eizo said:
    Am I the only one wondering if 1500Watt PSU is power enough at all?

    Under max. load this card hurles 360Watts per card!


    Where are you getting that number from? It doesn't even have the connectors to provide that much power without relying on a PCIe 2.0 connection as a requirement not an option.
  29. Hang on a minuite is it me or should this bit be raising the alarm bells.
    "Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?"
    XP only supports 2 GPU's dosent it ?
    Mactronix
  30. HUGELY interesting Post! Awesome setup man :) Looking fwd to hear from u further!
  31. Bump for greater justice...and news ofc !!
  32. mactronix said:

    XP only supports 2 GPU's dosent it ?


    No, and you're thinking about the render-ahead limitation in DirectX (not openGL), which doesn't apply to GPGPU apps the OP is using.
  33. TheGreatGrapeApe said:
    No, and you're thinking about the render-ahead limitation in DirectX (not openGL), which doesn't apply to GPGPU apps the OP is using.

    Slightly off topic.
    Ok i know you well enough to take your word for it Ape. :)
    Its just i have seen plenty of forum posts asking about tri sli on xp and everyone on 3-4 differant forums have said words to this effect.
    "Tri-SLI will not work in XP. XP is only capable of using 2 GPU's at a time. No matter what drivers are used. Even drivers coded by Jesus himself can't get past the 2 GPU limitation in XP. Sorry =o(, a Vista install is soon peeking over your horizon!"
    Can you send me a link or some such when you get time.
    Thanks
    Mactronix :)
    Ok guys and girls i will butt out now as i obviously am out of my depth here.
  34. Is this thread a joke?
    And 4 9800GX2?
    Are you boasting?
    Looks funny to me!
  35. kjoost said:
    I work as a Computer Scientist at an academic image processing department. In my research, I regularly work with nvidia Cuda (http://www.nvidia.com/object/cuda_home.html) for high performance computing on the graphics card (GPGPU). These are general computations that have nothing to do with rendering 3D scenes. For GPGPU, SLI is not required. Therefore, it seems possible to me to put four 9800GX2 cards on a single motherboard, for example the MSI K9A2 Platinum V2 (http://global.msi.com.tw/index.php?func=proddesc&prod_no=1395&maincat_no=1) that has four (physical) PCI-Express X16 slots with double spacing between the slots.

    When using CUDA, SLI is not used and each 9800GX2 is "seen" by the system as two separate GPUs. Therefore I am hoping to get access to eight separate GPUs by using four 9800GX2 cards. My computations can be distributed over these eight GPUs without requiring communication between the different cards (or between the two GPUs on each card).

    So far, I have not found any information from others who have tried this before. Cooling and power are obvious problems that can probably be dealt with. Any ideas on problems that I can expect on the software side? For example, will the (non-SLI) driver see the four 9800GX2 cards as eight independent GPUs? Or is there a maximum limit built into the current WinXP drivers for the non-SLI mode?

    Any comments would be appreciated!


    I think CUDA works with 4 pieces of 9800GX2 (8 GPU) without problem which is just highly depended on how you can modify the original driver by C language, however I just wonder if you need 8 cores CPU also in order to avoid bottleneck of computing. Except the monstor board D5400XS from Intel can support 2 CPU and 4x PCI-E 16 slots can meet this target. The only problem is no games could support CUDA coz CUDA main purpose is for performance computing not for performance 3D games.
  36. mactronix said:
    Slightly off topic.
    Ok i know you well enough to take your word for it Ape. :)
    Its just i have seen plenty of forum posts asking about tri sli on xp and everyone on 3-4 differant forums have said words to this effect.
    "Tri-SLI will not work in XP. XP is only capable of using 2 GPU's at a time. No matter what drivers are used. Even drivers coded by Jesus himself can't get past the 2 GPU limitation in XP. Sorry =o(, a Vista install is soon peeking over your horizon!"
    Can you send me a link or some such when you get time.


    Yeah, this is a limit for AFR under XP and DirectX.
    Here's a quick little blurb on it from the TechReport;
    http://techreport.com/articles.x/14284/2

    And the limit is for AFR so your could still do 3 frames ahead and effectively use 3 VPUs for just AFR, plus you could still have up to 6 VPUs rendering in AFR+SFR so A1+A2/B1+B2/C1+C2/A1+A2/.....
    My understanding is that this limit does not apply to OpenGL, as mentioned in this HardOCP article;
    http://enthusiast.hardocp.com/article.html?art=MTExNiwyLCxoZW50aHVzaWFzdA==

    If I can find more later, I'll re-post.

    However for what the OP is using them for, this is not an issue since it's not using them for rendering frames/frames ahead, but using them as co-processors, and basically he's just adding Gflops to his processing power.
  37. Thanks for that will read them later any more info apreciated.
    Mactronix
  38. Hi guys, it took a long time, but I will finally be able to report back to you on the result of this thread. Next Wednesday (May 28th), we will launch a website on our GPU system, with lots of information on the background of our application, system specs, benchmarks, etc.

    Stay tuned!
  39. Finally, I can answer the fundamental question of this thread:

    FOUR 9800GX2 cards: will it work?

    Check out our website http://fastra.ua.ac.be for the answer... ;)


    Thanks for all your help; we couldn't have done it without you :)
  40. Good Luck dude, all the best!

    PS: Are you Batenburg?
  41. Your are! Dr. K. Joost Batenburg!
  42. Good luck with your ptroduct, may it save many lives....and make you a few bucks and honor as well
  43. Yeah! I visited the site and the results are motherfragingly impressive!
    This may be going to be some really great breakthrough.
    Congratulations!
  44. romulus47plus1 said:
    Good Luck dude, all the best!

    PS: Are you Batenburg?


    Yes, that's me. :)
  45. DANG It !! [:zorg:2]

    I thought I was using my talents for Evil not Good; something like building a Supercomputer for a war mongering Banana Republic!! MMmmmm Bananas........

    Man I'm gonna get razzed about this at the next meeting of evil doers next Thursday. :sweat:


    __
    PS, sorry about the edit, undid it I think, mis-clicked the button. Or just being Evil. ;)
  46. Maybe if your photo/icon had a cape? heheh
Ask a new question

Read More

Nvidia Graphics Product