If you are going to use the stock cooler, you need to watch your temps very carefully. I would recommend not increasing the CPU voltage. The stock cooler has little reserve capacity as it is designed for use at stock frequencies in a "typical" (whatever that is) case.
If you are going to overclock, I recommend better cooling. The best deal right now is the CoolerMaster Hiper 212+. It's about $30.
I do not like to overclock graphics chips. I think they run hot enough anyway.