Between the 5770 and GTX260, I'd definitely choose the 5770. There is still room to improve the drivers, as evident with the new 10.3 which game the 5xxx cards a performance boost. The GTX 260 has been out a while and probably won't see much improvement. The performance is similar, the 260 does inch out most places, but who knows with the new drivers.
If you can swing a 5850 it would give you MUCH better performance on your new monitor with your shiny new machine.