Sorry, I haven't used d-sub in a while, just shooting out ideas. You ought to go through the standard stuff if you can, try hooking it up to multiple sources; try different monitors on your computer.
Having the V an H controls grayed-out, isn't atypical. It usually just means it's running on a digital signal, so it knows exactly where to put the pixels.
There's usually resizing options on both the monitor and the videocard, you should try playing with those (see how they work, don't just set it to what you think is right.)
What you're describing sounds like exactly what happens when you try to show a low res image on a high res screen that doesn't have a scalar. Some people do it intentionally, I did it to play Crysis.
You might have a scalar running on both the card and the monitor and it's getting confused.