Where do you live? The best place to buy them is somewhere like newegg or tigerdirect (among other places) if you're in the US. If you're in the UK or somewhere else - I'm not really sure which places are good for you, but I'm sure someone will have some more information.
If online isn't an option for you... there are little "hole in the wall" computer stores that should be able to help you out. Other than that, you can try Fry's Electronics for a decent price.
My estimate on prices (off the top of my head):
CPU: ~$279-315
Motherboard: ~$99-130
HDD: ~$80+ (Make sure the one you're looking at has a 16MB Cache and runs at 7200RPM)
RAM: ~$120-200 (Maybe less after MIR or if you buy 4x 1GB. I'm guessing 2x 2GB may cost a little more, but probably worth it if you want to add more RAM in later.)******
* Are you going to be using a 64 bit OS? If not - you really won't utilize all of that RAM. I'd recommend 2GB for a 32bit OS, but if you plan on upgrading OS within the year, I'd say it's worth the cost just to buy the 4GB now.