First off, I'm not sure how far you can overclock without maxing out your power supply. You are already on shaky ground. I would have gone with at least a 500 watt power supply in your case...
Getting your cpu to 3.6 - 3.8 should be possible without putting up a big fight.
Overclocking your graphics is a different story. Some cards oc a giant amount and some cards won't like going 5mhz over stock speed. You need to oc your graphics incrementally and test it along the way for stability... thats the only real way you can know the max your graphics card will go to.
PS: the best way to keep your cpu and graphics safe is by good cooling and smart moves. Don't go crazy on your voltages and you will be fine as long as your cooling is good.