Next stop should be a guide for your particular motherboard. Google is your friend.
You can also look for a 680i OC guide. Only difference between the two is that the 750i/780i was modified for Yorkfield compatibility.
Go through the guides. Then take your core voltage off Auto and set your memory voltage to factory recommended values. Change the System Memory Multiplier (or whatever your BIOS calls it) from AUTO to 2.00, 2.00B, or 2.00D - whichever you need to set the Memory Frequency to twice the FSB. Then when you increase the FSB, the memory clock will rise in in proportion with it. At an FSB of 266 MHz, your memory clock should be at 533 MHz. That seems low, but don't worry. As soon as you start pushing the FSB up, the mem clock will increase.
Download CPU-Z to check your FSB:RAM ratio. It should be a 1:1 ratio.
Warning - confusion factor between what the BIOS calls things and what CPUZ calls things. What the BIOS calls "memory frequency" is actually the memory clock. What CPUZ calls "memory frequency" is half the memory clock - DDR2 RAM, remember? It transfers two chunks of data each bus cycle. What you want in CPUZ is a 1:1 FSB:RAM ratio.
Intel's recommended max voltage is 1.3625 volts. Intel's absolute recommended voltage is 1.45 volts. Keep the core temps under 70C.