Next stop should be a guide for your particular motherboard. Google is your friend.
Go through the guides. Then take your core voltage off Auto and set your memory voltage to factory recommended values. Change the memory frequency to twice the FSB. Then when you increase the FSB, the memory clock will rise in in proportion with it. At an FSB of 266 MHz, your memory clock should be at 533 MHz.
Download CPU-Z to check your FSB:RAM ratio. It should be a 1:1 ratio.
You can run your FSB freq up to 333 MHz without outrunning your memory . And at 333 MHz, your core speed will be 3.0 GHz.