I was able to overclock my E8400 to 3.6 with stock voltages and stock heat sink. It really depends on a lot of different things so its difficult to say exactly what you will be able to overclock too.
Basically it would depend on your cpu stepping.. If your E8400 has an E0 stepping u might be able to get it quite far.. But if yours is a C0 stepping like mine u might need to pump alot of voltages when overclocking which well lead to an increase in heat and it will require a decent aftermarket cooler..
Most i was willing to do was 3.3ghz and my temps were hovering around 60. But with a decent new heatsink my temps dont go over 45c at 3.9ghz. Not every chip is the same though. If you really want the best out of your chip I would definatly ditch the stock HSF and get some decent cooling...