Is it it's Memory Bandwidth, Pixel Rate, Texture Rate, Floating Point Performance, Shading Units, Texture Mapping Units, Core Clock, Amount of Memory or Memory Bus?
What elements specifically give a higher frame rate?
Architecture, you can't compare cards based on specs. Only way you can is if they use the same GPU core, i.e. the 770 and 680, as both use GK104 Kepler, you can say the 770 is faster because its higher clocked.
Best way to judge speed is reviews, those websites like hwcompare are total crap.
Haha you're not going to like this answer but: All of those things. All of those things come together to give you the frames per second you end up getting. Certain games like increased memory bandwidth or shaders etc but overall more shaders and larger bandwidth is better. FLOPS is the only somewhat stable measure of performance but even then its not representative of real world performance. Also, +1 for guy above. Architecture is very important.