I am using a Gigabyte GA-G33M-DS2R mATX board in my rig. Check out the full specs. This board has a full range of OC settings (although I've only lightly touched a few of them). The chipset seems to be a little warm, but not too bad. I'm using a 7900GS, but may upgrade that soon (8800GT maybe?). It has at least as many connectors as the bigger P35-DS3R, including firewire. The X-QPACK2 case I used has excellent airflow, although I did get a bad power switch on mine. It will accept a standard PSU. I would suggest a modular one because there isn't lots of room in SFF cases for wads of extra cables. I like the Mushkin I'm using, but it is only Tier-3 so I might choose Corsair if I were building again.
Edit:
Oh, and as for cost, I re-used some parts, but I don't think I spent over $800.