Well...what type of PSU is it? I'm assuming you have enough voltage to power everything up, right? If so, you're right, one would assume it's your PSU. But unfortunately, it could be other things...such as your PCI slot on your Motherboard.
If you have a spare PSU, try that one...if you have a old PCIe videocard, try it and if it works then you know your motherboard is fine. Try and narrow it down as much as you can before buying new components (such as a new PSU).