I think GTX 460 768MB is definitely faster than 5770 1GB, about 10-20%. Its also more expensive. If price is not a concern, GTX 460 is a better option ($180). But if you have a budget, 5770 may be better ($130).
Other possible consideration may be the single card eyefinity configuration, if you ever want that, 5770 would be good.
Although games won't be able to be maxxed out, maybe games like dead space and a few others.