Probably is the PSU, but it could also be the motherboard or video card. Probably is not the CPU.
You can sort of check the PSU. Try to borrow a known good PSU of around 550 - 600 watts. That will power just about any system with a single GPU. If you cannot do that, use a DMM to measure the voltages. Measure between the colored wires and either chassis ground or the black wires. Yellow wires should be 12 volts. Red wires: +5 volts, orange wires: +3.3 volts, blue wire : -12 volts, violet wire: 5 volts always on. Tolerances are +/- 5% except for the -12 volts which is +/- 10%.
Pay close attention to the 12 volt lines (yellow wires). If you have a PSU problem, that's where the problem most likely is.
The gray wire is really important. It should go from 0 to +5 volts when you turn the PSU on with the case switch. CPU needs this signal to boot.
You can turn on the PSU by completely disconnecting the PSU and using a paperclip or jumper wire to short the green wire to one of the neighboring black wires.
http://www.youtube.com/watch?v=5FWXgQSokF4&feature=youtube_gdata
This checks the PSU under no load conditions, so it is not completely reliable. But if it can not pass this, it is dead. Then repeat the checks with the PSU plugged into the computer to put a load on the PSU.