You can NOT install DDR3 ram into a DDR2 slot.
If you get a new MOB that supports DDR3 it will perform much better.
The amount of RAM will be an improvement. The MHz will make no real difference in real world computing.
8Gb of RAM is a sweet spot that performs very well.
http://www.zdnet.com/how-much-ram-do-i-need-early-2013-edition-7000011139/
After 8GB it is not bang for your buck anymore.
If your MOB supports 8Gb of RAM you can get some great DDR2 ram.
Look on eBay for a bunch of good prices.
Happy Computing!