Motherboard: Go with an Asus P5Q (not the SE, Plus, deluxe, whatever). This is the basic version of those, its cheaper, very reliable, houses the Intel P45 Chipset, and tonnes of features to make your life easier.
CPU: Go for an Intel Core 2 Duo E6750. Its a very capable CPU thats also very, very afforable. If your willing to pony up the extra cache (lol, thats a good punn :3), you could get yourself an epic little Core 2 Quad 4-core CPU that wont go out of style for a good long time.
GPU: An Asus Radeon HD6570 should do the trick. (EAH6570) Low profile, low heat generation, direct x 11, will handle your applications with beauty and speed, and will render videos for output quite well. Also, should you ever dabble into *LIGHT* gaming, the card would serve you okay there too. Its also quite cheap when you compare it to other cards, and is of good quality.
PSU: Perhaps for quality, go for the Cooler Master M500-B power supply. Its about $45-$50, and gives you good connectors, a second fan to keep it going, and plenty of power. On the other hand, if your looking for cheap parts, try a Diablotek PSDA600. Its 600 Watts with just about every damn connector possible. Its about $30-$35 bucks, and gives you tonnes of power, includes a second fan, but I hear its a little unreliable sometimes. Some say its wonderful, some say its terrible. Its rated about 3.9 out of 5, so its not too bad, obviously. Maybe you should go for that.
Cooler: For your CPU cooler, I would say if you are willing to spend $35 - $40 on an excellent top-end air cooler for Socket 775, go for the Cooler Master Hyper 212 Plus. Excellent cooler with a massive fan and heatsink, and direct-touch 4 copper heatlines. However, if you really wanna budget, theres the $15 buck ones at Amazon.com that work about as well as a stock cooler. No copper, but a perfectly usable heatsink and fan regardless. It's also easier to install and is MUCH more low profile.
As for your case, your cables, and your storage, those dont matter terribly much. Perhaps, get yourself a DVD drive. Make sure its a SATA drive. Also, get a standard desktop HDD that runs at 7,200RPM and holds at least 120GB that also runs on SATA. Shouldnt be too pricy. Make sure to grab a couple SATA cables from ebay. Theyre about a buck a piece on there, and its a difference between having a nice liuttle computer and being stuck in the middle of butt-*** nowhere with no operating system accessible, and no linux-boot-from-cd from your DVD drive.
As for the case, http://www.directron.com/rc361kkn1.html . 'nuff said. Its cheap, it looks pretty nice, has good size and features, and is from a very good brand. Its compact, yet supports full ATX Boards. Always check the dimensions of the case and compare them to your hardware to make sure everything will fit. Good luck with your system!