Yes it makes a difference- esp in Games! Its easy to show you by looking at a celeron versus a P4 (if they had the same clock speed and FSB) - the celeron has the same instructions, just a much smaller cache- it simply dosent preform like a p4 (if a p4 can be said to preform).
If you want to casualy do nothing on your comp, get a celeron/sempron- they will do most things just as well as another chip. If you want to do anything serious at all, get a bigger, better chip. Infact, dont get anything. Just wait till conro comes out and get that. Or if your desperate to have something right now then get an amd chip. and overclock the sh1t out of it.