Yes your CPU is fine but it is not getting enough power to support that increase in clock speed, you need to down clock it til the problem stops and that will be your maximum overclock at stock voltage. To go beyond that mark you have to increase the voltage little by little. Do you have an aftermarket CPU Heatsink/Fan?
Ok, thanks for your advice. I am still quite concerned, since the OC tutorial on this site claims that you mostly start to see problems at 4.3GHz - 4.7GHz when you are at stock voltage. Now my clock is at 4.0GHz and I have not encountered this problem yet.
For the fan, I have the noctua nh-c14. I am planning on raising the voltages soon though to raise the clock to ~4.5GHz, I am just quite lost on what voltage to start with.