VGA: Analog signal. Limited by bandwith which prevents higher resolutions.
DVI: Digital signal. Usually can not carry audio*, **
HDMI: Digital signal. Can carry audio and video data.***
*Some DVI solutions can carry sound when signal is converted to HDMI. Both NVIDIA and ATI GPU's, using their DVI-HDMI adaptor, can carry audio over the connection. This is not typical of DVI connections though.
**There are various forms of DVI connections, each with a slightly diffrent pin layout. The most common reference is to the digital connection used to hook up from a GPU to a moniter, although other DVI connections, including an analog DVI, exist.
***HDMI, in part to much higher bandwith, can carry an audio signal alongside a video signal. For Video, HDMI and DVI signals are exactly the same, and can be converted back and forth with ease. For audio, HDMI is also the only solution that can carry high end 7.1 sound to sound systems, thanks in part to its higher bandwith, which has farther made HDMI the new standard for Home Theatre systems. (Spdif, the previous digital connection for sound, can only carry compressed (DD/DTS) 5.1 sound)