I seem to have difficulties OC'ing the 7770 vapor-x, if I increase the either GPU or memory clock my FPS drops but if i leave it at default (1.1Ghz-core/1.3Ghz-memory) i have better FPS (like 10-15+ better FPS than if I try to OC the card, even by 10mhz) which is kinda weird for me since thermals are very low (less than 60C max running anything)
My questions > is the graphics driver the problem, or just my rig... dont know if my rig is the bottleneck though (worked fine with my trusty old - gts 250)
The Ghz edition is already overclocked and the vapor-x is overclocked by 100mhz over the standard Ghz edition so to overclock your Vapor-x card you will have to add voltage as you overclock.
There are three Ghz edition cards and they are , 1000mhz , 1100mhz and 1150mhz. I don;t see anywhere in your post that you have added voltage with your attempted overclock. When overclocking either the cpu or gpu you can only go so far with changing the core clock and then you have to start adding voltage with the core change , it's not much just the next increment. If you still get the same fps problem then there is a bottleneck and it would most likely be the cpu. However you do have a odd setup of ram and it would be better to have a complete set from one manufacturer , plus you are running on an older computer tech with DDR2 and socket 775 and your trying to put brand new tech (7770) with it. That's most likely why the GTS 250 ran well with it because it was also old tech.