I'll be shopping in-store at best buy, and I can afford $500-$600 in upgrades. What should I do to start building a more futuristic system, yet ensuring my other components are still compatible. Should I start with a new motherboard so I can run PCI Express 3.0? Or is this not necessary.
Are you stuck with best buy? because that SEVERELY limits your choices. They have poor selection and bad prices. Is there a Microcenter or Fry's nearby? Or can you get a newegg giftcard type thing? (I assume your challenge is credit card based).
I think you are ready to retire your LGA 775 motherboard. You should be able to jump to a modern system for a $200 cpu, $100 mobo, $80 memory, $60 CPU cooler. but only if you can purchase online. This would be an extra $100 at worst buy...
Based on your specs a mobo / processor / ram update probably makes the most sense in terms of future-proofing your build.
RAM is cheap and as long as you have enough of it, it generally won't tend to affect your in game performance. Any DDR3 module should do (maybe get 2x4GB).
I'm seconding deadlockedworld's recommendation. Grab an i5-2500k on a Z68 mobo with 8GB of Ram. Get a solid Heat sink / fan and your system will be blazing. If you weren't limited to best buy you could even test out an SSD on your system for your OS / applications. Remove the pagefile / hibernate file and you are good for quite a few games!