Get a gigabyte mother board as they seems to be more compatible, this one is quite new and you can get it for 135$
Gigabyte LGA 1155 DDR3 1600 Intel Z77 HDMI SATA 6Gb/s USB 3.0 Mini ITX Motherboard GA-Z77N-WIFI
As for the GPU you may get an ATI 6870 as it is one of the best GPU that can run on a hackingtosh out of the box(no patches or anything), buy a Saphire or a HIS IceQ version (HIS has a better reputation to be quiter, mine is very silent when running funs on 50%)
Even when a hackingtosh support integrated HD 3000 I have found so many issues so it's a good idea to get an descrete GPU card.
For overclocking I guess you may want to get a sandy bridge processor as i5 2500k but not sure about the gigabyte overclocking capabilities (I currently own an asus but I will be migrating to gigabyte).
Check out the bitfenix prodigy case, it seems nice and it's not so small so it will be more versatile to fit a big GPU and a big CPU cooler.