8X Zotac GTX680 : Problem getting 8th GPU to work??? Please Help

[davesura]

The Build

Motherboard: Tyan FT72-B7015
Processor: Dual Xeon X5680
RAM: 144GB
Hard Drive: OCZ Vertex III
GPU: 8X Zotac GTX 680 2GB

I have installed every driver version provided by NVIDIA and I'm continuing to have problems initializing the 8th GPU. The GPU's are all operational and can be moved to any slot and function properly. There is no OC'ing with any of the GPU's. Windows recocognizes all 8 GPU's however it is disabling the one on PCI Bus 12? Computer managment says there is a problem and giving Code 43 (worthless info) and disabling hardware. When installing different driver versions the PCI Bus that gets disabled can switch to different GPU's. This makes me believe that it could be a driver issue with NVIDIA. Is there any BIOS settings that could prevent 8 GPUs from operating properly? Does anyone have any suggestions that I could try? I'm going to round up 8 Tesla C2070's to see if those work. It could be a problem with the new GTX680.

 

[Mousemonkey]

Code 43 is a dead hardware error but if it moves around and the same card works in a different slot then you could well be on to something, maybe the 680 being a desktop card can confuse the drivers once you go over the five GPU's that in theory would be the max (Quad SLi + PhysX).

 

[Sunius]

Did you try that last GPU on its own? Maybe it is faulty?

 

[davesura]

I currently have 7 working. It's just the 8th card that is giving me trouble.

 

[davesura]

Did you try that last GPU on its own? Maybe it is faulty?

Yes all GPU's are working properly. I can swap any GPU to any slot and all function up to 7 in series. As soon as I try the 8th GPU windows disables one of them. Typically its on PCI Bus 12 but again with different driver versions I've seen windows disable different cards ...ex PCI Bus 132 etc.... This variable varies depending on the NVIDIA Driver version????

 

[Mousemonkey]

The 680 had its GP compute abilities nobbled did it not? Perhaps that's the reason why you can't run 8 or the mother board can't power 8 slots or 2 of the slots share resources?

 

[davesura]

The 680 had its GP compute abilities nobbled did it not? Perhaps that's the reason why you can't run 8 or the mother board can't power 8 slots or 2 of the slots share resources?

Hmmm.. Nobbled not sure? I know we can run 8 GPU's on this motherboard. We added 3 power supplies to the server to sustain the load and it can handle it without a problem.. There is something with the settings but I cant figure it out. I'm pointing the finger at NVIDIA but was posting here to see if anyone had suggestions I could try?

 

[Mousemonkey]

You said earlier that you could source 8 Tesla cards, have you done so yet?

 

[davesura]

You said earlier that you could source 8 Tesla cards, have you done so yet?

Not yet. I'll go do that real quick and report back with my findings. I have to go borrow some from a couple different workstations.

 

[Mousemonkey]

If 8 of those run OK on that mobo then all questions are answered IMO.

 

[FunSurfer]

The Build

Motherboard: Tyan FT72-B7015
Processor: Dual Xeon X5680
RAM: 144GB
Hard Drive: OCZ Vertex III
GPU: 8X Zotac GTX 680 2GB

I have installed every driver version provided by NVIDIA and I'm continuing to have problems initializing the 8th GPU. The GPU's are all operational and can be moved to any slot and function properly. There is no OC'ing with any of the GPU's. Windows recocognizes all 8 GPU's however it is disabling the one on PCI Bus 12? Computer managment says there is a problem and giving Code 43 (worthless info) and disabling hardware. When installing different driver versions the PCI Bus that gets disabled can switch to different GPU's. This makes me believe that it could be a driver issue with NVIDIA. Is there any BIOS settings that could prevent 8 GPUs from operating properly? Does anyone have any suggestions that I could try? I'm going to round up 8 Tesla C2070's to see if those work. It could be a problem with the new GTX680.

Did you try the RAM tweak?
http://social.technet.microsoft.com/Forums/en-US/w7itproperf/thread/265b7dff-3063-4056-871f-1dee4a505359

 

[davesura]

Did you try the RAM tweak?
http://social.technet.microsoft.com/Forums/en-US/w7itproperf/thread/265b7dff-3063-4056-871f-1dee4a505359

Yes I have, still has no effect on the system. I have ran into the slow system problem with other cards and that process fixed that issue however this is something I have not experienced. The system is physically disabling the 8th card saying there is a hardware failure when there is not.

 

[nforce4max]

http://ob-security.info/?p=274

 

[Mousemonkey]

http://ob-security.info/?p=274

Oooh!

There was some issues initially with getting all 8 cards to work, but once the BIOS was configured correctly both Windows 7 and Ubuntu 64 were able to detect all 8 cards.

 

[davesura]

http://ob-security.info/?p=274

Now all 8 GTX680 are working by enabling the onboard VGA in bios however the nvidia control panel and CUDA dll's are not running. Is there a way to force NVIDIA control panel to intialize drivers. I do not use the graphics cards for video, they are specifically for processing data sets. I just need CUDA to work, no video..

 

[Dangi]

Sometime ago someone created the Fastra II, one computer with six GTX295 to compete with real supercomputers and being cheaper and faster rendering images.

Whoever they couldn't get the Fastra II to work due to some BIOS problems, they couldn't get all cards to work at the same time, they contacted Asus and some with some teak in the BIOS they got all cards to work.

Maybe is your BIOS that doesn't let you run 8 cards at the same time.

 

[davemaster84]

Sounds like a faulty card, or maybe the sli bridge however it works in that configuration.

 

[zloginet]

Leave onboard video enabled in BIOS but put the video order to try pcie first? Some boards have that option... That may allow windows to default to you nvidia control panel first?

Also, does windows device manager show both cards under display adapters? If so, disable your onboard video and then log off and log back in, that might also default to your nvidia control panel then?

 

[Onus]

Disable anything else you don't need; firewire, serial/parallel I/O, or anything else that might be hocking an interrupt or needed I/O address.

 

[s3anister]

wicked thread, I love to see pics of this beast once you get it up and running to your specs...

good-luck.

I second this motion.

Too bad I don't yet have experience with 8x GPU configurations... :lol: otherwise I'd try to help.

 

[davesura]

Alright I loaded Ubuntu Desktop 12.04 and updated all NVIDIA drivers and all 8 GPU's are functional. NVIDIA Control panel is showing all eight cards, CUDA Cores, Temp etc. This is a Windows issue. hmmmm. I have some engineers at Zotac looking into the problem and hopefully we can find a solution. If they run fine on Linux I don't understand why they don't on Windows.

 

[nforce4max]

I know that windows isn't the best os around when it comes to certain features and applications. Could be that the os isn't able to maintain more than 7 gpus at once or the forceware for windows has some odd limitation. Cool as it is but there were a handful of simulators made by Quantum 3D back in 2000 that supported up to 32 gpus (3DFX VSA-100) but sadly DX limits everything to only 4 gpus for modern games.