I'll put it this way... I hadn't taken the HSF off my cpu for nearly 5 years. I always cleaned it while in the case. Then one day I decided to take it off, because I had some thermal paste that I bought for my two laptops. Dropped 4C ilde, 6C load by replacing the old paste. Pretty impressive for 5 minutes of work.
Anyhoooo... the stuff gets dried out and not as effective even when you're not messing with it, so yes if you take off the HSF please do clean it off and put some more goop on it.