There are a number of possibilities here, faulty graphics card, faulty power supply, faulty motherboard. Are you sure the computer is not booting? or is it that you can't see the computer booting where as in fact it is? You can judge if it is still booting even though nothing appears on the screen by looking at the hard drive LED.
On most modern motherboards when you plug in the graphics card it will disable the internal graphics, so connecting to the motherboard VGA connector is a waste of time when the graphics card is fitted. Saying that check the BIOS to see if there is a setting for this.