If you move from a 5770 to a 6850 or 6870 you may not get enough of an improvement to really see the difference. The table suggests 3 tiers to see a difference. (Benchmarks will precisely measure the diff, but you won't feel it during gameplay). Consider instead HD6950 or GTX560ti or better. Prices on those may come down as new hd7850 comes avail.
Now your questions:
"compatible with powersupply? " 450 watts is more than enough. The i7-860 and MB and everything non-video draws peak well less than 200 watts. The 5770 draws 108 max (google card name and 'tdp' to get max power for a card, roughly). The 6870 draws max 150 watts. 6870 will not even warm up the 450 watt PSU. This article will make you comfortable with that. They measure max at the wall of 295 watts which at 80% efficiency is only 236 watts used of the PSU max. http://www.guru3d.com/article/radeon-hd-6850-6870-revie...
"compatible with motherboard?"
If you can run a 5770 your MB will support a 6850 or 6870 no problem. Just check your case vs. the length of the cards. Both should fit no problem. As you get to the higher end cards length may become an issue so check it.
Edit: aside, the desktop review calls out a 500 watt PSU. The PSU link is for a nice 450 watt PSU. I assumed the 450 for above.
I think your photo shows a 6 pin but it's a bit fuzzy. You can compare your 6 pin to the photo in the adapter link above.
Aside: Your old 5770 card is still $100 new, and worth something to somebody. It's low power so will fit in most/all systems without a PSU upgrade. You can make a friend by giving it away or you can ebay it. If you put card in your basement (like i do) then make sure it's in an antistatic bag or box. The box the hd6870 video card came in will have an antistatic blister pack you can reuse.
Nice card! I see the card comes with molex to 6 pins adapters if you end up needing them.
I've never had a problem with Newegg. I did one ugly return for credit when i needed to replace a SF1200 based SSD that wouldn't detect in my laptop with an Intel 320 based one. Newegg covered me above what they said in their return policy.
not sure if this is a real problem or me not knowing what im doing, but i cannot disconnect the 6 pin connector from the 5770. how exactly am I supposed to do this? I've tried firmly pulling on it and it just doesn't come out.
am i missing something really obvious here or did gateway do something to prevent upgrading?
Ok, I have the new card installed and everything is running fine! although there wasn't a MAJOR performance boost, it was still quite noticeable. for example, BF3 was running on mediumish settings with 30-35 fps on my old card, now I'm running it with everything on ultra (except anti-aliasing) with 45+ fps.
im also surprised at how cool and quiet the card runs in my poorly ventilated case. it runs at around 75c under load with the fanspeed at 50% (sounds very quiet) and my old card ran at around 90c under load with the fanspeed at 90% (sounds like a jet taking off)
anyway, thanks for all the help tsnor! I hope this thread will help other FX6840 owners searching for help on google.