The 6770 may be hitting it's vram limit at ultra, turn it down to high, and try again.
I am using 6770 and BF3 on it @ ultra setting. Never face a problem. Though my res is low(1280x1024) and fps is low(25-35). So vram is not a problem i think.
Another thing is hitting vram will not do that type problem i hope. Cause i experienced vram limit when playing Dirt 3 online at ultra setting. Window performance tools report to lower my visual setting. It will cause low fps in game(22) but not crash.