Intel i5 750
standard psu (350W?)
I'm looking to upgrade my system to be a little more game friendly on high/ultra settings for sc2, diablo 3, dow 2 etc... I've done a little research but I'm still a little new to this and don't want to get something that won't fit, overheat, be excessive loud.
A friend recommended an HD 5850??
You are going to have trouble finding a 5000 series card at a good price. They are mostly phased out. You'll first need a PSU, I suggest a Corsair 650TX. 750 also. Make sure your Dell system uses a standard ATX PSU. I'm guessing it does because it's not a slim case, but make sure it's the standard size.
How much do you want to spend? A 7770 will be fine, and if you can spend more for a 7870 go for that. If you have even more money get the new Nvidia 680. However... you could post a picture of the inside of your case so we can see how it is layed out for suggestions on room for these cards. The 7770 is the smallest of these.
Note: If you go high end on the GPU, you might want to change my PSU suggestion to make sure you have enough PCIE power connectors for their demands.