Clean out the CPU fan and the case, make sure the airflow isn't restricted and see if that helps.
Cleaned them out. No change. Set them to full ball - no change. Ended up cracking the case open and have an external fan blowing at the CPU -> no more alarm so far (with a/c on at least).
Thanks for the advice. Would still prefer to find a way to actually set the alarm threshold higher or off. But this PC is 5 years old now, time for an upgrade anyway