I'm leaning towards a bad video card. Had a 8800 that did the same thing, ran fine on 2D, soon as any type of 3D work was started it rebooted or locked up the PC totally.
I'm giving aford's idea 10% chance of working :-) especially it if happens in every 3D application.
Another thing, but less likely than a bad video card is Power Supply, then RAM. First thing is to swap out the video card, if issues go away, there you go. If PC still reboots, you need to swap out some more components.