For the normal user, there's no real downside.
Heck, I used my mini-tower with a Micro ATX motherboard as a sort of gaming PC for the longest while. Just ran a single good quality video card and other components and did some case modding to add better airflow for the case.
It got ugly when I stuffed my CM Hyper 212+ in the poor thing. Had to slice a few holes in the side XD.