Need help overclocking for the first time

First up, on your build - hopefully you have a GOOD psu. That's absolutely necessary as you need to keep the voltage stable - and overclocking increases stresses exponentially as the voltage usage and cpu frequency goes higher (and out of spec). Also, there is no hard and fast rule as to how high a particular processor will go, each one is different. That being said, usually for Skylake I've seen normal OC's going up to about 4.6ghz.

The general limits on voltages and temps are as follows:

Skylake - Volts: 1.40v NO higher. Temps: 75C on average. Little higher/lower fine, constant higher temps though degrades your CPU lifespan. Usually 75C is nice and safe for daily usage.

Software needed: Prime95 for stress testing. HWMonitor for keeping tabs on temps during overclocking.

1) turn your multiplier up by 1.
2) stress test. If passes with no errors, and temp is below 75C go to step 1. If fails, or temp exceeds 75C go to step 3.
3) if voltage is at 1.40 or temp is 75C, you're done. go to step 5.
4) turn up voltage .01V. Go to step 2.
5) turn multiplier DOWN by 1
6) stress test. If passes, go to 7. If fails, go to step 5.
7) Finished.

Your average CPU is tested to be super stable and run as cool as possible, so it's clocked at a particular voltage and speed for efficiency. With better cooling, it can run faster, but it'll also need more voltage. More voltage increases stability at higher speeds, but also causes higher temps. So - you're going to creep up on the max frequency at the lowest voltage based on YOUR cooling which is better than stock.

So you're going to increase the frequency at normal voltage until it needs more, then you'll increase voltage, then check the frequency and increase again, moving both up until the limits are met. Once you hit your thermal limits at that chips particular voltage you'll turn it down a notch to be stable. Of course, this is all general OC'ing guidelines, and your specific instructions depend on the motherboard bios.

So - use this as a guide, and at your own risk. You should be able to get it to 4.3-4.4ghz easily before having to increase any voltages. Again though, each chip is unique in it's makeup which makes some a little better, some a little worse.

How much should you overclock your CPU to: That's all up to you and it also depends on your cpu cooler which is the hyper 212 which is decent enough for a medium overclock on an i5-6600k. But you will have to play around with the voltages yourself at the set frequencies its all about stress testing your system after putting in the overclock, and if it crashes turn down the frequency or bump up the voltages its pretty simple

You overclock it and keep it that way. You don't want to go into the bios everytime and reset the overclock do you? that's just silly ;P So yes overclock it and keep it there unless your temps are too high.

it is very worth it to overclock your GPU, GPU's have huge headroom for overclocking and you should have no issue overclocking a GPU just make sure you use the right program depending on what card you are using.