Try this: turn it on so it works. Shut it down with the case switch. Now, reach around to the back of the computer and either turn off the PSU or unplug the AC power cord. Wait 10 seconds, turn on or plug the cord in.
Then try to turn on the computer with case power switch. If it comes on, wait for boot and OS load to stop. Then turn it off. Repeat two or three times. If the PC always comes on, you have a flaky PSU.
"Flaky" is good in a pie crust. Not so good elsewhere.