Can you put the old 8800GT back in and see if it works fine? If it works, then RAM the 5850.
Also, plug in both DVI and VGA and manually try setting it to DVI via the OSD on the LCD.
Well, the 8800gt is broken, except for safe mode, but I'll give it a try. As for your second idea, I already tried that, it wouldn't detect it the monitor. However, it I hooked it up through my Sony surround sound reciever (via HDMI), it did work digitally (at 640x480...)