First off, you'll get around an 85% performance increase from your GTX 260 (almost double!).
Second, your PSU is just fine.
Third, an HD5770 wont be limited by a PCIex8
Can you tell me why it won't be limited by the pciex8? Is it because the 5770 isn't fast enough to limit it? I never really understood how that bandwidth stuff worked on motherboards.