You are trying to set a resolution/refresh rate higher than your monitor can handle.
Try going to Advanced settings on the Video control panel and check "hide modes your monitor cannot display" (or whatever looks like this) and set everything up again.