The HD5770 could run on your PSU if it has enough amps on the +12volt rail. The GTX260 will not. The GTX260 is slightly more powerful but at the moment costs a lot more. HD5770 has DX11 and eyefinity.
For the reasons above I would select HD5770
The value of the 5770 in particular is clearly not going to be in its performance. Compared to AMD’s 4870, it loses well more than it wins, and if we throw out Far Cry 2, it’s around 10% slower overall. It also spends most of its time losing to NVIDIA’s GTX 260, which unfortunately the 4870 didn’t have so much trouble with..
I'll pay 20% more for a 10% performance increase but I won't pay 33%....I'd buy the 5770....assuming of course you are committed to 1650 x 1050 gaming for the life of the card. At 1920 res, the entry point is the 5850 / 470 . Both have a "dollars per frame" cost of about $9.00