A 8600Gt will run that display perfectly fine.
Is the card newly installed? maybe update the drivers? Do you have a spare DVI cable to try? That seems odd. 1920 x1200 on a large display will have larger pixels, so the edges may seem a little jagged, but that's about it. Very strange. Which DVI port is it plugged in, I believe on the Port 1 has HDCP support. Either way you could try the other port.