I went from MSI 560Ti to MSI 660Ti. And I feel like the difference is big. Now, I play BF3 on Ultra and Skyrim at High. BF3 looks awesome and never has lag (only internet lag)
You could sell a Used 560Ti for $170. So, MSI 660Ti at $300, is really like spending $130. So far I think it's worth it.
The 670 would be better, obviously, but at 1080p, the improvement wasn't justified for me. Just look at benchmarks.