First and foremost, do a complete clean Uninstall of the AMD drivers as they are no longer needed and will only interfere with the proper operation of the Nvidia card. You can use a program called "Driver Sweeper" and it's free, just boot into Safe Mode and then run the program and select the AMD drivers to be uninstalled completely. Then you shut down your PC and swap the cards out. I also clear the CMOS for when I do a major hardware change such as a CPU, GPU, RAM and such. Just take out the CMOS battery and wait for about 30 seconds and then put it back into the motherboard. This clears the BIOS and should help with making your card work.
Then install the GTX 770 and make sure you connect both the PCI-E power connectors the GTX 770 requires, your PSU should have them. Then boot up your PC followed by downloading the latest Geforce drivers from Nvidia.com and you should be good to go. Also, if the DVI output on the card is still not functioning, try connecting the card to a VGA monitor or your HDTV via HDMI.