Any motherboard that has a PCIe x16 slot (whether 2nd gen or 3rd gen) will be compatible with the GTX 1060 or RX 580 (as both cards have a PCIe x16 interface).
http://www.tomshardware.com/answers/id-3281149/gtx-1060-intel-h61-chipset.html
http://www.tomshardware.com/answers/id-3225555/gtx-1060-compatible-h61-motherboard.html
https://linustechtips.com/main/topic/779460-does-asus-h61-m-cs-motherboard-support-gtx-1060-6gb-gpu/
The specific GPU choices takes into account the CPU you are using (i5-3450), which will not hinder the powerful GPUs' performance in most situations (except, possibly, in very very CPU-intensive games).
Just make sure your current PSU has ample wattage and PCIE power connectors depending on the GPU model that you will choose. For the GTX 1060, usually, a ~450W PSU will suffice. For the RX 580, usually, a ~550W PSU is recommended.