HD5770 and GTX260 are similar in performance. It does not make sense to upgrade only motherboard and ram just to get DDR3, a full CPU/mobo/ram makes more sense.
Looks like the issue you had was the ver2.1 card in an older 1.x slot. Any new Nvidia card are version 2.0 and should work fine.
In regards to the 5770 bein similar performance to the gtx 260. i had the 5700 of mine working breifly and i had 0 issues in game until my pc rejected it from the slot issue. but my 260 has reeeeeal bad slow down issues that make some games impossible to play. and as for the upgrade.. i already have a card/ram just no mobo/cpu to upgrade to