It will not affect performance negatively. Vs what you have now, but...
It will probabally be better to just run the 4GB at 800, and remove the old ram entirely.
The reason for this is because when combined, all ram has to be ram at the same speed. The high speed ram can downgrade, but the low speed ram can't normally upgrade, so it will run at the lowest speed. For most people, 6GB at 667/5300 will run slower than 4GB of 800/6400. If you are actually using more than 4GB RAM, which 95% of people don't, then you should install all 6GB.