It will not harm the power supply as amuffin says, it is just coil whine.
You may try messing with power saving settings to see if you can stabilize the sound to some extent(but it comes at the cost of power use).
I have found coil whine when running some power supplies at very low loads.
If you can RMA it may be worth a shot, the only issue is just like video card or motherboard coil while, it can be hit and miss. This means you may get another unit with the same/worse sound(or if lucky one with none at all. its a hit and miss game).