I'm leaning towards a bad video card. Had a 8800 that did the same thing, ran fine on 2D, soon as any type of 3D work was started it rebooted or locked up the PC totally.
I'm giving aford's idea 10% chance of working :-) especially it if happens in every 3D application.
Another thing, but less likely than a bad video card is Power Supply, then RAM. First thing is to swap out the video card, if issues go away, there you go. If PC still reboots, you need to swap out some more components.
if under OC condition :
- Clear CMOS : takeoff Power from AC + Cable monitor form CPU
- Check compabilitas RAM with memtes86+ if have error replace New RAM
- Uninstall soft make OC CPU & GPU
- install again your old driver (cd bundle with VGA)
if default condition
go safe mode uninstall your driver VGA make reboot next install VGA driver (cd bundle with VGA)