I had a QX9650, which is cut from the same silicon as your chip. I needed 1.35v to make 4Ghz. I think i needed to keep the FSB at 333Mhz to do it though, 400Mhz fsb needed a bunch of other tweaks. So up your voltage a little bit, and change the multiplier up one tick and see what happens.
the native FSB of the QX9770 is 400MHz, the QX9650 is 333MHz. My QX9770 hit 3.6GHz on stock voltage (9x400). I had to up the voltage on the NB chip on my motherboard in order to have the ram settle in at 1600 (400 FSB). My guess is that you NB chipset and ram can't handle your overclock. what is your ram set to in your bios? this is a Dell machine? what motherboard do you have? you will probably only be able to overclock by upping the multiplyer...ie.., 9x400 etc. you won't be able to overclock the NB and RAM. Also some motherboards have problems with all dimms occupied (4x1 GB) and overclocking
it might be easier to set the multiplier to 9 and have 9x400 (3.6). you can change the fsb strap for odd multipliers in BIOS. (set strap to 5:6 for 333 to 400) so your cpu has a 400MHz fsb...your ram and motherboard is 333MHz fsb native. So it's probably blowing out your ram and motherboard's northbridge if you're trying to bump up the RAM to 1600MHz. Also 4 dimms are much harder to overclock than 2 dimms. If you want 4GB, try 2x2GB dimms of DDR3-1600. Usually you have colored ram slots...for my motherboard the blue slots are for faster 2 dimms. you can pull 2 out and try to overclock.
Anyway, best way to overclock might be just to overclock the CPU and forget about overclocking the FSB. You'll have to leave the FSB at 333MHz or underclock to 200MHz (800 after the 4x memory multiplier) and deal with the overclock with the strap setting.