Well, unfortunately, if you can, I would return everything except the HDD and RAM and start over.
The PSU is trash. This is 1000% the most important part of a PC. A low quality one can easily fry everything. Use ONLY the brands I listed. It should be replaced ASAP.
The CPU is a bit weak for a mid+ range GPU. It is not meant to be used with a GPU. It is for HTPC and office computers, not gaming rigs.
If you replace the PSU, you could add something like a gtx660, but I would definitely not get anything more powerful than that for this CPU.
Something like a gtx750TI would be great here. Should be able to still get medium-ish settings out of the computer.