The amount of memory on the graphics card is independent of the system memory. There is no limit imposed by the system to the memory used by the graphics card (although bottlenecking could be an issue depending on your CPU).
The limiting factor would be your 300W PSU, I'd think you could upgrade to a GTX750Ti without having to worry about upgrading your PSU