The single slot was notorious for noise and terrible thermals. If modding is to much for you then go with a new card. A 5770 is aging and the GTS450 is like a lot of cards out there and is a under performer for it's specs. So look to a 6850 if you are lucky to find one at a reasonable price or a cheap but reliable GTX460. I got ATI and Nvidia and Nvidia has much better drivers. Got S3, SIS, 3DFX, Trident, Cirrus Logic, Matrox, PowerVR, and several more brands under my belt so I got a good grasp when it comes to cards unlike most who learned only on modern machines before making the next step. The 5770 isn't the best ATI card out there but it is 6-7/10 depending on build quality. The GTX460 wins out due to more features and performance for the price. 7.5/10 Quality of modern cards leaves much to be desired so there isn't even a card that is 9/10 in my book.
If modding, the best quality cooler is a Zalaman VF1000. I wouldn't recommend this if I didn't spend $50 for one my self. Get any grade hack saw to cut away the power vrm end of the stock cooler as you will need it. Mount the new after market cooler and vram heatsinks that come with the cooler. Attach the vrm part of the stock cooler that you have cut away and there should be two screws to mount it into place. Be sure to not use the included compound as it is no good. Use arctic silver 5 or any other compound of your choice. If the temps improve but not enough to your liking then pair it up with a pci bay mounting fan or fan kit that blows air into the cooler. This will improve the performance of the cooler greatly and is more than enough to air cool even hotter running cards such as the 8800gtx.