Try and find out. I have the 6350, which is just a factory overclocked 6300 to 3.9ghz (4.2ghz with turbo boost on), in a garbage case and it gets up to its max suggested temp of about 63C under full load. But that's with stock CPU cooler and only 1 case fan.
You can get a Hyper 212 Evo for about $30 and then I would say you are golden to do some overclocking.
Your best bet though is to just try it out and see what your temps get up to. I use HWMonitor to check my temps, and you can use AMD Overdrive if you want to try and overclock using software, or go through the bios if you know how or would rather do it that way.
Also check out speed fan to control your fans and rev them up if you want to try and increase airflow.
The FX-6300/6350's overclock very well from everything I have heard. I have not overclocked mine yet but I want to do some upgrades to my case, CPU cooler, and PSU before I try overclocking.
Give it a whirl and monitor those temps and see how it goes. I definitely would not let my CPU go over 65C. I usually start getting nervous anything over 60C.
Good luck!