You will see quite a large difference between your P4 and the Celeron E3200. You'd see more of an increase with the AMD quad, but really it's based on your budget. It will certainly be more than a 100% increase in performance - your current chip is more than 5 years old. If you're looking to spend more than $150 for the processor, I'd suggest going for the i5 series (LGA 1156) and spending $150 more on CPU, Motherboard and RAM just to ensure that you'll be able to upgrade in a few years at a lesser cost. LGA 775 is close to the end of it's life, and anything you buy now for that socket isn't going to be upgradable without a motherboard + memory upgrade along with it.
The onboard graphics should perform the same no matter which processor you drop in. That is to say, not very well. But if the extent of your need for graphics is limited to lightweight games and streaming video, it will do the job just fine. You can always update your graphics card later, and a good $90 model will take you leaps and bounds above onboard video. But the graphics card isn't the question at hand, so I'll back off that one =).
I think you're shooting a bit high at 4gb memory, BUT, you'll have shared memory with the integrated graphics, and at $30 for the extra 2gb there's no reason not to get two 2gb DIMMs.