Currently I installed 2GB (2X 1GB DDR2 PC2 5300) and I have two additional slots in the GIGABYTE GA-MA785GM-US2H motherboard. Can I install an additional 4GB (2x 2GB DDR2 PC2 6400) in the other two slots? Would it effect the performance negatively?
It will not affect performance negatively. Vs what you have now, but...
It will probabally be better to just run the 4GB at 800, and remove the old ram entirely.
The reason for this is because when combined, all ram has to be ram at the same speed. The high speed ram can downgrade, but the low speed ram can't normally upgrade, so it will run at the lowest speed. For most people, 6GB at 667/5300 will run slower than 4GB of 800/6400. If you are actually using more than 4GB RAM, which 95% of people don't, then you should install all 6GB.
^+1 It will run, but it will run in Single-channel, not dual-channel, because one memory is unpaired.
The difference between single and dual-channel is maybe ~5%. Not much difference. It's still up to you.