The GTX 770 consumes about 30W more than the 9800 GX2. Your PSU can definitely handle *one* GTX 770 if it could handle *two* 9800 GX2s. If your PSU has 850W or more, you can even add a second GTX 770.
The 9800 GX2 and GTX 770 are both around 10.5 inches long, so you should be OK from that point of view too.
It has a 1000W PSU, so I guess I should be good Watt-wise even with a second GTX 770. Thanks for your reply.