There are 3 heatsinks for that. from the top they all look alike unless you can see one without the shroud, or get a bottom view. There is a solid aluminum one which is the least efficient for 65W CPUs, there is one with 4 copper pipes and a copper base plate, and there is one with 6 copper pipes and the copper base. None of them are very expensive surplus. The best place to start for cooling is the heatsink so the heat leaves the CPU faster, if that happens the intake fan can run slower due to lower CPU temperature. The expansion slot blower will help GPU temperature greatly. I needed one for the GTX750TI.
You can get someone to test your GPU for you before you mod it. I actually own a BTX Mini Tower, and a Zotac GTX1050Ti, I've also installed GTX750TI 2 slot card, and R9-285 ITX card. I also have a R9 Fury Nano I haven't tried yet. So I have a pretty good idea what's involved. Any GPU upgrade for a BTX system will probably require more skill than snapping Leggos together. The Zotac Mini is as close to a drop in 1080P solution as you're going to get.