the thermal protection for the cpu should've kicked in to prevent catastrophic damage to the cpu and components...by kicked in...I mean the system should've shut down if thermal protection kicked in which from what you posted...that's exactly what it did.
From there...since you rebooted successfully and was using the machine to play games...there should be no damage incurred.
Then you stated that you removed the stock heatsink and fan and then you reinstalled it without...changing the paste? or removed the paste and didn't put any on the cpu when you reinstalled the cooler?
Depending on which it was...thermal protection circuitry and sensors should've kicked in at some point....one a bit quicker than the other...and shut down the machine again.
You are now experiencing idle temps of 40C on the stock Heatsink and Fan or HSF?
First when you overclocked...I would say your temps at your overclocked speeds overwhelmed your stock HSF...thus creating a thermal overload and the thermal protection bsod the machine...the stock hsf is woefully poor as an air cooler at stock speeds much less overclocked speeds...so I looked here
and it seems that 40C isn't out of the normal range for idle on these processors.
I guess the next question would be is 40C way higher than before you began overclocking?
Also...I would say that the reason it's bsoding and acting up is because you either
A. didn't apply the paste correctly, or used too much, or used too little.
B. or you haven't got the cooler locked down fully or you smeared the paste when reinstalling the heatsink and fan.
C. You didn't mention what TIM paste you used so if you didn't get a quality paste and apply it correctly you have created a thermal interface that isn't performing as well as the original paste between your ? stock heatsink and fan? and since the stock HSF isn't able to deal with the heat from the cpu at stock OR OCed speeds now...I would say the paste you applied hasn't "cured" yet. All thermal paste has a "burn in" period where over time the interface performs better after curing and achieving lower temps...but usually we are talking anywhere from 20hrs to 200+ hours to achieve optimal conditions for the lowest temps and efficient performance out of the HSF and paste!
If it was me...and I truly believe you are having thermal issues here...I would invest about $35 and purchase this...
The absolute most bang for the buck and this thing performs like mad...
I believe you will see your issues resolve themselves and rather quickly especially when attempting to game and this "Could" allow you to overclock successfully. I say "could" because each processor and motherboard and HSF combo performs "uniquely" or differently...the amount "if any" of achievable overclock is dependent on many different factors..but "HEAT REMOVAL" has to be effecient and effective no matter what...and the stock HSF isn't worth a hoot!!
Recheck that the HSF was reinstalled correctly and that you have good contact between the cpu top and the bottom of your HSF with a "very thin" layer of paste covering the entire top of the cpu and that the top of the cpu and the bottom are in "full" contact for even and efficient heat transference from the cpu to the cooler!!!
One of the best ways to spread the paste in an "EVEN AND PAPER THIN" layer is to use a single edged razor blade to lightly apply the paste over the entire top of the cpu heat spreader..."the top of the cpu itself".
Sorry for the lack of details! I will try and address your queries
1. My motherboard is a Gigabyte GA-P55-UD3 (Rev 1.0). I also flashed the bios to F9 (the latest) in order to overclock.
2. I removed the CPU heat sink and fan in an effort to clean what looked like a huge build up of dust (it wasn't as significant as I thought). I then, in a rookie move, replaced the CPU heat sink and fan back on top of the CPU WITHOUT removing the old thermal paste and reapplying a new coat of thermal paste.
3. The thermal paste I used was: Cooler master HTK-002-U1 Thermal Compound. I applied a pea and spread it evenly all over the cpu, then put the heat sink immediately on top and clipped it back into place. I did the best I could to make the CPU stick tight.
4. The paste has only been on for 2-3 hours. I gave it half an hour with the case on its side in order for it to stick well.
I also have I7-860 and one time my stock cooler just died on me and I didn't realize it until when I'm playing BF3 and my computer would just shut down automatically. I checked my temp and it was at 99C! So I think if it overheated it would've shut down your computer for you like it did for mine. I have a Gigabyte UD4P mobo.
Your paste doesn't EVER get "sticky" it's supposed to remain supple and greasy...if it begins tacking up or getting excessively sticky...it's beginning to dry out. The paste just acts as a thermal interface between the top of your cpu and the bottom of your cpu cooler.
You did it exactly right when applying the paste and correctly spread it out very thin...that's the proper way to do TIM for the best results.
It may take a bit to achieve the best performance.
As far as your shutdown...you should take a look at your "event viewer" to see what caused the shutdown.
they will describe how to help determine what event caused the shutdown...it's not always immediately what the messages mean but it usually points one in the general direction of what's going on...read the error messages and root out the specific item that created the "Hard Lock" bsod.