There's no reason a GTX [strike]1070 [/strike] 1080 Ti wouldn't work on that motherboard. Or any motherboard with a PCIe x16 slot (which is virtually any consumer mobo).
Not familiar so I’ll wait for someone who is to pass judgement. The 1080Ti is a 250~ card though and the 6700K which won’t be OC’d as you have a H110 board is under 100 so you should be fine as long as te PSU has the right amount of PCIE 6+2 pins the card you buy needs
Given that you won't be overclocking, you can probably get by with a ~$30 cooler (maybe even less?). Or you could go with a Ryzen 2600(X), which come with a cooler.