Alright, I think I'll dabble here and see what I can contribute.
According to this: http://www.tomshardware.com/reviews/gaming-graphics-card-review,3107-7.html
(Tom's Hardware Graphics Card Heirarchy), the GTX 650Ti is slightly more powerful than the Radeon HD 7770GHz Edition, at 1920x1080 (I'm assuming that's going to be your resolution), and they can be found for less than $150.00 USD, like the Radeon HD 7770GHz Edition. Both these cards are good, and have their pros and cons. Both have similar memory configurations, and around a similar core, but from what I have seen, personal experience, nVidia drivers tend to roll out and patch more, more frequently than AMD drivers. The thing is that AMD does a good job with their drivers, allowing them to continue to deliver better and better performance in modern games.
At this point, the Radeon HD 7770GHz Edition or the GTX 650Ti are your best bets. Either card will bring the performance you want, and are significantly better than the card your have now. Ultimately, it comes down to personal preference, AMD or nVidia.
*EDIT: Your PSU seems powerful enough, but can you give a brand and model for the PSU you have. It's the best way to see if it will properly hold up, even if it is a 650W PSU.