There is no way that power supply is going to be powering anything above a 7750. From what I see, the 12v rail supplies 18A, the 7750 requires 16A so it should be barely enough considering your other components.
Another thing is the form factor, is this a regular sized Inspiron 560?
Don't get the 650 Ti Boost, your CPU would bottleneck anything above a GTX 650 or 7750
Considering your current position, the Radeon 7750 would probably be your best choice
^+1 on everything, just to add a bit of info for you, 650Ti Boost (140W), 650 (65W), 7770 (55W). Throw in 200W for everything else on your machine plus some overhead, 300W would manage a 650 or 7770.