PSU : $ 30 only - Corsair 400w.
You can also choose Antec, XFX, etc & you won't need anything above 450.
ODD : Go with Samsung / Sony /LG. I'll go with this $23 Sata LG DVD burner. The one you have is an IDE, & it's just a CD drive in any case.
That should do it. Seeing as you were using the Dell Dimention, you probably wont need a discrete GPU anyways, but if you did, then this is the one for you for $46 only - Radeon HD 4650
If you need a OS you need to spend $100 for Windows 7 HP 32 bit. You can also go with the 64 bit version, but i'd stick to the 32 bit because of possible driver compatibility issues with some older hardware that you might have lying around.
If you change all your older hardware (webcam etc.), you might wanna go with the 64 bit. But then again you'll have to upgrade the RAM (costly!!!) 'coz for 64 bit, 2GB is the minimum req, & the recommended is 4, unlike the 32 GB version, for which the min. is 1 & the reco is 2.
If you do decide to change the CRT monitor to LCD, go with a 20'', 16:9 one with minimum 50000:1 Dynamic Contrast ratio, 5 m/s refresh rate & 400 cd/m2. Below those specs, your screen will look washed out & fast moving objects will leave a trail behind. Stick to LG / Samsung etc.
It's only around $400 with OS & excluding monitor (which you can upgrade next year - maybe) & gfx card (you can make do with Integrated gfx for now). Of course if you can find some combo offers with these same / similar products, it might be even more cost effective.