Yepp, it's what VeTeRen ThoR said. I'll just add a few things though:
For your current computer setup as it is now, your PSU (Power Supply Unit) is sufficient. It can provide a proper electrical current of 250W. 250W is quite weak, but since the hardware inside your computer is not very demanding it really doesn't matter. However, if you plan to upgrade to a system that can actually do some budget or at least entry level gaming, you'll need a new PSU with a higher wattage. I recommended that you get 500W as it will give you enough power to supply a rig with a Radeon HD 7850 and an equivalent processor. If you notice, the PSU I linked you to is of "80 Plus Bronze Certified". This means that the PSU has been certified to run at at least 80% of its specified wattage without wasting the energy as heat. Comparatively, a PSU without an 80 Plus certification will waste more energy as heat and may not supply enough energy to your system. This is more for your future knowledge in case a few years down the road you decide to buy a brand new PSU. For now, the 500W PSU I linked you to will do well.
However, the other issue is that your current CPU is very weak and it will bottleneck your graphics card for a number of games if you decide to upgrade your graphics card past what it already is. Like VeTeRen ThoR said, bottlenecking is when your CPU can't keep up with your GPU (Graphics Processing Unit, a.k.a. Graphics/Video Card). It's a little more complex then that, and in reality the level of bottlenecking varies from game to game. In games that really make use of your GPU a lot more than your CPU, bottlenecking may be negligible. Such games can include Crysis 2, Crysis 3, Battlefield 3 (surprisingly), The Witcher 2 and so many more. Conversely, bottlenecking can be extremely bad in certain games that make a large use of your CPU or are "CPU bound". Such games can include Crysis 1, Crysis Warhead, Borderlands 2, Skyrim, and again, so many more. Basically, the best way to ensure that your system handles each and every game thrown at it well is to make sure its parts are balanced well.
Now we get to the motherboard. I looked at your CPU and its pretty weak. Unfortunately, your motherboard only accepts the outdated AM3 socket type (Not to be confused with AM3+). Socket types are layouts on motherboards to allow certain CPUs to be accepted. An AM3 socket type will only accept an AM3 CPU. The AM3 socket is outdated, and because of this not only are AM3 CPUs no longer manufactured, but the ones you can find on eBay are either weak or overly expensive (because they've now become collector's items). So what I recommend is that you buy a new Motherboard as well. Basically what I'm telling you is, your computer setup as a whole is in dire need of a new setup. Too many parts are simply too old and weak. The good news is, upgrading your PC as a whole can be relatively inexpensive. With shipping and mail-in rebates included, you can expect to pay a little over $200 for all this, but it'll be a notably more efficient computer in the end, not only for gaming but for anything you throw at it:
PCPartPicker part list /
Price breakdown by merchant /
Benchmarks
CPU: Intel Pentium G2020 2.9GHz Dual-Core Processor ($61.99 @ Amazon)
Motherboard: ASRock B75M-DGS R2.0 Micro ATX LGA1155 Motherboard ($57.98 @ SuperBiiz)
Video Card: PowerColor Radeon HD 7750 1GB Video Card ($63.98 @ Newegg)
Power Supply: Corsair Builder 430W 80 PLUS Certified ATX12V Power Supply ($22.98 @ Newegg)
Total: $206.93
(Prices include shipping, taxes, and discounts when available.)
(Generated by PCPartPicker 2013-08-28 13:40 EDT-0400)
I know I originally went with 500W, and you wanted an HD 7850, but 430W is still more than enough and you'll notice the difference between your HD 4200 and this HD 7750 right off the bat.