Currently, Nvidia chipsets are regarded as poor to us OCing enthusiast. There are also multiple problems with the board, as reviewed by Anandtech. Apparently it's better now, but there still are issues:
http://www.anandtech.com/weblog/showpost.aspx?i=460
The fastest way to solve the issue is to go with the Intel chipset. If you do not want Crossfire, P45 is your best solution. If you do however, X48 is the way to go, although it has been reported that it's OCing capability is much lower than the P45.
The other way is to spend a lot of time getting your system stable. With a Vdroop so large, I'd say start with a Vcore of 1.400 in the bios. It should droop down to 1.300v~, which is below the max safe limit of 1.4v. Then set the ram to it's rated settings, linked or unlinked. After than, start playing with the motherboard settings, increasing or decreasing the voltage and such. Perhaps try using different multipliers, it could be that the setting you're trying to do has a hole in it.