RX 580 / 1060 6 GB would've done the job, but...
first, price is volatile right now, i suggest u stick with your old GPU for a bit longer
second, it's not worth upgrading for, the performance gain isn't really worth buying (replacing a GPU for only 30-40%ish increase?)
if u want real upgrade, check Vega / GTX 1070-1080 instead (of course i can't recommend these GPU either coz it's way out of your budget...Vega being out of stock [rumours said that on Q4 the custom-cooler edition is going to come, which means the thermal problem should be solved / at least minimized by them], 1070's volatile price, and 1080's MSRP)
Suggestion?
Clean up your case to help the airflow
if it isn't enough, opening the side panel should help cooling it a bit...