My new system is a Q6600 @ 2,4Ghz on a Gigabyte EP45-DS3R, with CPU cooler, the Asus Triton 70 and a Powercolor ATI Radeon HD4850 graphic card.
Included in the mobo software it's an application called "Easy Tune 6" to allow some overclocking under Windows and also as an hardware monitor.
At 2,4Ghz, the default speed (266Mhz x9 ratio), the CPU temp is 25C, although the system temp is 51C (the graphic card is 81C). I simply tried to increase the FSB to 333Mhz to obtain the 3Ghz, but the system froze! I'm a newbie when it comes to overclocking, so I used the Basic mode of "Easy Tune 6" that would only allow me to change the FSB. Could someone help me get to the 3Ghz? I don't want/need to go any further. I have the latest BIOS.
Don't use the included software to OC -- as you found out, it just causes problems. Change settings directly in the BIOS instead. The forum stickies include good guides on how to do so.
First, though, find out what part of the system has that 51C "system temp". That, and your high graphics card temp, strongly suggest that you've got a big case ventilation problem. Make sure you have at least two 120mm exhaust fans in the back/top of the case, in addition to your power supply's fan.