Get a HD7870 and see how it goes. Since you say your current graphics card overheats your PC, I'd get this one, which exhausts its heat: http://www.newegg.com/Product/Product.aspx?Item=N82E16814161404
(assuming it will fit in your case).
Edit: If you are still unsatisfied with performance, then you can look into a CPU/Mobo upgrade, but I'm doubtful you'll need that. Having more RAM than you need isn't hurting anything. Look at your case cooling too; make sure you have decent airflow, generally front to back.