Can I safely disconnect auxiliary power from motherboard?

Welcome Newcomer!  

^Pretty soon it will quit whining, your PSU is beginning to fail. "If it were me" I would shut the PC down and get a good replacement PSU pronto.

Replacing a PSU now beats replacing your: MOBO, CPU, GPU and/or pretty much everything it's attached to...  

There's no auxillary 12V power. The 12V is completely turned off when the computer is shut down or put into standby, but the 5V standby supply, on the violet wire of the main power connector, continues to operate, unless the AC is disconnected. Get a power strip or battery backup, and shut it off to stop the coil whine.

Coil whine can be from loose windings or from failing capacitors changing the duty cycle of an oscillator or causing the driving transistors to switch less abruptly. Capacitor failure is not always visible.

^ Yeah and on a 5+ year old PC it's a PSU needing replacement.

Duh - "windings or from failing capacitors"

Alternatively soon I can help @404zero determine whether their MOBO, GPU, CPU or all of the above is toast.

So I guess my only option is to replace the power supply.

The question is, can I replace this BTX power supply with an ATX?

I've heard mixed opinions as to whether or not this can be done.

Thanks again!

The BTX Power Supply Unit is the same as a standard 24pin ATX PSU. Its physical size may vary depending on the size of the chassis.