GTX650 Ti by miles. But like people say, the Boost is hugely superior still (around 40% faster on average).
You could always check into the 7790, provides more performance than the 7770, and it looks like it is going to be used in the next series of consoles, so you could probably get away with running console graphics settings for a while in this next generation.
A stock speed GTX 650 Ti's performance basically equal to the performance of a stock speed Radeon HD 7790. Therefore, the HD 7790 is the GTX 650 Ti's direct competitor.