How to find your core voltage depends on your motherboard.
Anyways, first things first, get a core temperature monitoring program such as Core Temp or SpeedFan. Then get Prime96. All these programs are free, so don't worry about paying anything for them.
After that, restart your computer and launch the Bios. The button to do this depends on your motherboard as well, but it'll show on the motherboard splash screen when starting up your computer.
In my motherboard's BIOS (Gigabyte Z87X-UD4H), the Core Voltage and multiplier options were on the main page, so no searching was required. Other mobos may require some searching, you will probably find it under a tab called "Tweaking" or "Processor". I don't know for certain though.
As for the actual overclocking, it's fairly simple. For example, 35 Multiplier = 3.5GHz, 43 Multiplier = 4.3 GHz. The core voltage required varies from processor to processor, some are better than others and require less voltage to remain stable. An easy method of overclocing would be setting the core voltage to 1.25V and the multiplier to something like 43.
Once you do that, exit the BIOS and log in normally. Turn on whatever temperature program you have and launch Prime95. Start the torture test, and let it run for a few hours. As long as your temps don't go above ~80 degrees celsius and your computer doesn't crash, you have a stable overclock. Getting a higher overclock involves simply going back into the bios and raising the multiplier.
If your computer does crash, you need to either lower the multiplier or raise the voltage. As I said, as long as temperatures don't go above 80C you are fine to raise the voltage higher.
I was able on a 4670K to get 4.5GHz stable with 1.25 volts (I think). Your mileage may vary.
I think I covered everything.
And trust me. It's a lot easier than it sounds.
1. Temps don't go above 80 while stress testing.
2. Stress testing. Let Prime95 run for at least a few hours each time. I've had overclocks turn out not to be stable after 6-7 hours.
3. Multiplier = Core Clock.
4. Higher core voltage (VCore) = Higher potential multiplier and more heat.