Knowing what country you are in would help us price parts better. Right now the 650ti is only $5 more than the 7770 in the U.S., so the 650ti wins here.
You're right in thinking you'll need more RAM. You'll want to get at least 4gb in the system. Hopefully your machine uses DDR2 (which is expensive enough), not RDRAM.
$110 MSI 650ti
and assuming you have 2x1gb sticks in your machine right now,
$27.80 Crucial 2x1gb DDR2 800mhz