The risk of damage is extremely small, but you should know what you're doing. You must monitor temperatures to be sure you're not running too hot, and you should make sure your Vcore doesn't rise too high. I prefer to keep my overclocks pretty mild, so I don't ever increase my Vcore, and I don't do anything that requires resorting to extreme cooling solutions.
On newer motherboards, you can often overclock without knowing anything about it. In my BIOS, I set my base clock up from 200 to 220 MHz, which gives me a 10% boost in CPU speed. I've been able to go as high as 30% over stock on my overclocking, and I'm far from being an expert.
As for the E4300 chip, there have been reviews where they tuned the chip for a 60% increase in clock speed, at stock voltage. This is why I'd be plenty comfortable putting a 33% overclock on the chip as I plan to, because I know even at that speed I'm not coming near to pushing the limits of what the chip can do.
So, if you want to dabble in overclocking, I'd suggest you look through your BIOS settings and see what's available. Be sure you have your motherboard manual handy in case you need to reset your settings back to standard. Start small and work your way up. If you get hooked on overclocking, you may want to consider upgrading your PSU if you need to, as well as your heat sink. Moderate overclocking gives you a lot of extra value for your dollar. Extreme overclocking is a hobby unto itself.