In real world conditions a CPU can never run too cool and you don't have to remove the stock cooler later on when you upgrade.
There are a couple points on the planet that are cold enough to potentially produce a cold bug
. Not that anybody is running their computer on the Antarctic wasteland during winter, but still. Real world!
But really, "too cold" happens at around negative 60c to negative 80c, and then it's certainly not damaging the chip, just causing it to crash until it warms up again. Go ahead and put the cooler on.