The 4GB model is designed for SLI setups and multi-monitor configurations. When the 4GB of VRam is unneeded though (either for the GTX 670 or 680) it has been shown to perform a tiny bit slower than the 2gb version: http://webcache.googleusercontent.com/search?q=cache:C6-aMd3xkQgJ:www.pugetsystems.com/labs/articles/Video-Card-Performance-2GB-vs-4GB-Memory-154/+&cd=2&hl=en&ct=clnk&gl=uk
It's likely due to 4gb being way too much for 1080p, and just dragging the card down (by using up resources on memory management or the like) rather than helping it. You definitely shouldn't expect to see any performance increase at 1080p with the 4gb model.
How much of a frame rate difference is there though, is this would only explain maybe a 5% difference? I assume you've checked you're playing at the quality settings? What games is this in?
@matto17secs He said 1080p