2500k Idle/Load Temps: Should I OC?

Should isn't the right word there. You can OC. And yes your temps are plenty good for it. But only OC if you need more performance in apps/games that can make use of it.

I guess I'll see how it works out for me after a while, as I just bought the computer. If needed, I will do it down the road then.

the whole point of a 2500k and a CM Hyper 212+ is to overclock

those are great temps

Core Voltage in CPU-Z is the cpu voltage

for a overclock of about 4.5ghz you should look to keep it roughly 1.35v and less than 1.4v

its possible to get 4.8ghz to 5ghz if its a good chip --but keep an eye on temps and voltage

^^ Overclock tht CPU!   4.5ghz is a great OC

Temps are fine, go for it  

fantastic temps. and yes you SHOULD oc. why get a k processor if you arent going to overclock it?

you actually lose features (feature) getting a k processor, so best you take advantage of the bonus

You definitely have the thermal overhead for a substantial overclock. The question is should you? You can push 140 watts easily with an over-volted (~1.35v) and over-clocked (4.5 GHz) (mcnumpty23's good numbers) i5 2500k, and if you are running the PC 24/7 that can add up. Then there is cost of evacuating more heat. Since it is a "k" part, I'll go along with the rest and recommend overclocking, if only to see what numbers you can push. If you can use the speed (GPU bottlenecked by CPU, but that is doubtful with an i5 2500k), sure!

Edit:
I have to remind myself Its only one processor (I bitcoin mine / BOINC on the side and I often undervolt/clock the CPU, and do the opposite for the GPU, if only to save a few $ and few degrees). Go for it, and if you suddenly feel hot, drop it a few hundred MHz.

The great part about overclocking the 2500k is there is probably more overclocking information on this CPU than any other in history. The 2500k and 2600k have been hugely successful and there are literally hundreds of sources you can use to get the fastest, stable, overclock your CPU can handle. So that's the good news. The not so good news is you can damage your components if you're clumsy enough.

The most important consideration when it comes to overclocking, especially if you're new to the concept: it takes time to get it right! DO NOT RUSH! Take your time to research, make small adjustments and be sure to test your changes before moving on.

Set a goal for your OC and performance. While it's fun to see what the limits are, staying within reasonable voltages and temperatures will prolong the life of your components, not to mention help to keep you stable. Typically on a 2500k, you can reach frequencies anywhere between 4.2-4.6 GHz by making MINIMAL changes to your settings. In fact, you should be able to easily get into that range by making only changes to the vCore, and even then the "Auto" setting works for a lot of people.

If you're willing to put the time and research in, and you're able to set a reasonable goal for your OC, I think you'll have a lot of fun overclocking your CPU and it's a pretty safe process. It is a powerhouse of a CPU, by anyone's standards, and I think you'll be amazed at how fast your system is when you get into the 4.2GHz+ range. The forums on Tom's are full with people using this processor, so don't hesitate to ask if you run into unanswered questions. Good luck to you!!