[Help] Have to wait hours after an power outage to turn on computer

This problem has been getting worse and worse over the last few months. When it first happened I could just reset my surge protector and turn the computer back on. Then I had to switch off the PSU and unplug for 10 minutes, now it takes up to 3 hours until I can turn my computer back on. The last time this happened I completely rebuilt my computer outside the case to make sure grounding wasn't a problem. Tested the RAM by switching them and using 1 at a time. Used the paper clip trick to make sure my PSU was still working. I pulled the motherboard battery out for a couple minutes and put it back in and still no luck.

What I would really like to know is which computer part is failing.