Some overclocking questions from a complete beginner

I'll answer as best as I can.

1.) Coretemp, Realtemp - either of these to monitor the core temperatures of your processor. These should also be set with the maximum temperature your core can go to. Your motherboard monitoring software doesn't measure this close to the core.
Prime95 - use this to cook your processor for a long time to ensure stability, if any of the threads stop it means your overclock is unstable.
Intel Burn test - since you have an AMD processor I don't know what mileage you'll get from this but it works an intel processor harder than prime95, heats it up further and will typically throw up errors sooner.

2.) 4GHz seems like a good place to start, but others with more knowledge and experience of the FX4100 may give you pointers.
3.) It is worth it. Your whole system will get a boost in performance. I can't quantify that in gaming as it will depend on the game and how the rest of your system meshes together.
4.) Use coretemp/realtemp as a guide, your looking to be no closer than 20-30 deg to the tjMax.
5.) Your system will be a bit noisier than it was before, only you can determine if this is acceptable or not once you've done the overclock.
6.) No. Overclocking increases the heat output of the processor and you need a beefed up cooler to take the heat away. For the most part the stock AMD cooler struggles to deal with the processor at stock speeds and should be upgraded anyway.
7.) Learn how to manually adjust the vcore of the processor, this can be used to lower heat output at lower overclocks and increase stability of higher overclocks. If you increase the vcore of your processor too far you will break it.
Take note of your current vcore and fix this in bios, you should be able to do a certain amount of overclocking before having to adjust this further.
You can go hunt down youtube vids if you want but I find overclocking forums a better source of information as you get the discussion between different people about "safe" overclocks whereas youtube will give you one person's point of view (much like this post).