I'll elaborate a bit on the Best Answer, to clear up some areas he glossed over. To install a new GPU, it is indeed as easy as fitting it in and connecting the power and video cables to it, as said above. That's pretty much it on the hardware side.
However, before you install the new GPU, you need to first uninstall the integrated graphics' driver, usually named something like 'Intel HD Graphics' or similar under 'Uninstall Programs' in the Control Panel (For Intel CPUs, that is). Then, after installing the GPU and connecting up the cables, you need to go to your GPU manufacturer's website (Either AMD or Nvidia, not the OEM like MSI or ASUS) and download the corresponding drivers for your GPU from there. If you don't do this, you'll run into all sorts of issues, and games will run very poorly.