You should really use the DVI ports, both on video card and monitor. But they are more expensive, but should have been included with the monitor (both VGA and DVI). And like rolli59 said, only one cable! Either DVI or VGA.
DVI = Digital signal, and will get the most out of your VC 5850.
And like rolli59 said, did you plug in a PCI-e plug from your PSU into the card?