The 460 is a better card than the 6770 even with it being the 768 version. Is it over clocked or is it standard.
Also what is the rest of the machine going to be ? What CPU are you using with these cards ? What resolution ?
Where are you shopping ?
Generally speaking, the higher the resolution you play at, the more VRAM you want on your video card. For a 1920x1080/1920x1200 1GB of VRAM is enough. Above that then you want more VRAM. I would say that for 1680x1050 512MB of RAM would be adequate, but this resolution is kinda like on the borderline as to what's better, 512MB or 1GB?
VRAM is not everything though, because the GPU must be fast enough to take advantage of the available VRAM. For example, the Radeon HD 5670 was a pretty good mainstream gaming video card when it 1st came out. However, there is virtually no performance difference between the 512MB version and the 1GB version. The GPU core was simply not fast enough to use that extra 512MB of RAM.
I agree that the GTX 460 768MB will give you better performance than a Radeon HD 6770 1GB.
A GTX 460 769mb is approximately 25% faster than an HD6770 at 1080p on average at reference speeds. The GTX 460 can also OC a ton which would make the difference even larger.
If you are planning to SLI keep in mind that the GTX 460 768mb is already quite rare so you'll probably need to get a used card of ebay to add a second when the time comes.