AS Saint said- you do need to Upgrade the Mobo as well to use DDR3.
However- the GTX260 is just fine as a card, but it is significantly overpriced for its performance. It will perform very similarly to a 5770, but the 5770 is only $150 and it has DX11 and Eyefinity capability. The 5850 is a great card as well, but its in the ~$300 range, which may be a good bit more than you are wanting to spend. Also- do you already have 2 GB of DDR3 or do you have 2GB of DDR2? In the latter case, you will not be able to use it with the new DDR3 you are getting. Either stay with your current Mobo and just get 2 more GB of DDR2, or upgrade the mobo and get 4 GB of DDR3.