Next stop should be a guide for your particular motherboard. Google is your friend.
Go through the guides. Then take your core voltage off Auto and set your memory voltage to factory recommended values. Change the System Memory Multiplier (or whatever your BIOS calls it) from AUTO to 2.00, 2.00B, or 2.00D - whichever you need to set the Memory Frequency to twice the FSB. (Asus uses something called "straps".) Then when you increase the FSB, the memory clock will rise in in proportion with it. At an FSB of 266 MHz, your memory clock should be at 533 MHz.
Download CPU-Z to check your FSB:RAM ratio. It should be a 1:1 ratio.
Keep your core voltage under 1.5 volts and keep your load temps under 70 C.