Personally I've never had a psu with lights on the back. However it seems that the yellow light would indicate if something is wrong (atx connector uplugged), so that is normal right? Do you have the manual? You can look it up.
Seems logical to get another psu and try it.
tried another power supply and it worked ok but wanted to make sure the problem could not be rectified on old psu - didn't want to throw good psu away....