Start from the beginning:
Your power supply has plenty of Watts and amps, I ran a HD4870 from a weaker Hiper PSU without issues.
First, ignore the current card, get the 9600 working, so you can fully update Windows and DirectX.
With that done download Driver Sweeper and install it.
Redownload the Nvidia driver, you may have a corrupted download, it is rare but can happen.
SET A RESTORE POINT!
Power down and put the GTX260 back, making sure it is fully seated and locked into position. Connect BOTH 6 pin cables, making sure they are fully seated.
Now reboot and go into the BIOS and reset it to factory defaults, save and reboot.
If the lockup still happens:
Reboot into Safe Mode and uninstall any graphics drivers you find-ATI or Nvidia.
Run Driver Sweeper but remember: This type of software is not perfect and it can damage your system, so check everything and deselect any items that look wrong. This is why you set the Restore Point!
Reboot, you should boot normally this time because no drivers are installed but the graphics settings will be poor. Tell us if this fails.
If all is well, install the drivers you freshly downloaded and reboot.
Post back with the results, please.