I would not overclock with stock cooler, it will get quite hot and may shorten lifetime.
When you get your CPU cooler, which I'll 100% suggest a Cooler Master Hyper 212 EVO as it is the best one for the price ($35), I would OC it to 4.5-4.6Ghz.
Good temps are below 65 degrees, though it varies
Hope this helps you.
Thank you, this is the cooler I was planning on buying.