Mix of RAM pairs
Currently I installed 2GB (2X 1GB DDR2 PC2 5300) and I have two additional slots in the GIGABYTE GA-MA785GM-US2H motherboard. Can I install an additional 4GB (2x 2GB DDR2 PC2 6400) in the other two slots? Would it effect the performance negatively?
It will not affect performance negatively. Vs what you have now, but...
It will probabally be better to just run the 4GB at 800, and remove the old ram entirely.
The reason for this is because when combined, all ram has to be ram at the same speed. The high speed ram can downgrade, but the low speed ram can't normally upgrade, so it will run at the lowest speed. For most people, 6GB at 667/5300 will run slower than 4GB of 800/6400. If you are actually using more than 4GB RAM, which 95% of people don't, then you should install all 6GB.
I would just install the new sticks in the first 2 slots(yellow) and the old ones in the last(red).
This should give you dual channel(but it will be at the speed of the older sticks). Despite what many say, you can use different sized sticks(in pairs) and keep dual channel working.
Then you can try with just the new sticks(they will operate faster)and see if you actually notice the speed difference.