YES. You should ABSOLUTELY be upgrading your CPU - that's going to be huge, as that CPU is not meant for being hefty at processor-intense tasks. Unfortunately you can't upgrade that without also upgrading your motherboard.
I highly suggest you save up until you can afford a low-end i5, a motherboard, and a decent graphics card, since you want to be running those games at high / ultra settings.
The good graphics card will get you the settings, but it'll be expensive.. and you HAVE to have a decent processor if you want to actually stay at a high framerate - otherwise it's going to be as laggy as heck.