I assume you have a PCIe card slot. The AGP slot is getting too old to find cards for now (and the CPU will bottleneck even if you find a fast AGP card)
The 3870 looks much better than the 8800GTS.
Only the 8800GT is better for games. But the 3870 has HDMI and 10.1 so it's more 'future proof'
The 8800GT and 3870 are the only gaming cards in the world that matter now.
If not serious about games, the HD 2600XT or 8600GT is good enough for Sims2, PGA Golf, even Oblivion on mild settings.