Very few PSU's have user adjustable outputs.
HardwareMonitor has problems showing the correct voltages.
What kind of PSU is your old one? How does it act in your system?
In fact, how does your system act with both old parts?
Install the old parts and test your system. If good and your old PSU is at least a 500 watt unit, install the new video card and test.
You may be able to verify that one of your new parts is bad.
Basic rule of upgrading: if possible, upgrade one component at a time.