hmm so your CPU is an LGA775 socket, and as far as i know you can fit the GTX 770, however it will get bottlenecked because the CPU and motherboard are considered to be quite old. so you wont get the most out of the GPU.
I dont know what your situation is, but if you have about 500-600$ you could get a new CPU, Motherboard, and GPU which will last you the next few years.
basically, you shouldn't go spending money on any newer GPU, unless you get a newer, faster CPU and motherboard.
hope this helps.