I think it's a pretty universal truth that making things smaller has a cost... be it performance, price or both. If cost is your primary concern and space isn't an issue, I think it'll always be better going with a full ATX setup. You're much more likely to encounter thermal issues with a smaller case.
It won't be much cheaper, and you'll lose some potential to upgrade (let's just say that the case will be too small for a lot of video cards, for one). The trade-off is very similar to that of desktop vs. notebook - you're essentially trading performance for portability, and portability is often at a price premium as well (unintential alliteration).
If you want a small computer that's easy to lug around and you're not that concerned about upgrading, go for it, but to answer the question, you won't save much money.