This is completely normal, caused by a tiny spike in CPU usage.
So, I actually decided to test different fan positions and what not.
I actually managed to fix my random spikes of temperatures.
It was due to the fan in the back of my case not actually pulling air and pushing it outside of the case. It was instead doing the reverse effect of that. It was blowing air onto my fan that was pushing air and it was sorta messing with the airflow from the push/push from the heatsink.
Thanks for the responses guys, I'm fine now.