Oh yes I highly recommend going modular. What a difference it makes in ease of installation and also the cleanup.
As for ram, always try going for exact same ram that you already have. If you get two different sticks and they're now different speed, all 4 have to run at the lower speed (whichever set is lower). Also, two old and two new ones may not necessarily play nice together (as you're discovering). If you're going to have them all 4, do not try to overclock them in any way. Run them at their default basic speeds, and once you install them, reset your bios (from within bios), just to make sure everything is back at default values. Also when you're installing them, try different configurations. Like slots 1/3 for your old memory and 2/4 for your new. Then try 1/2 and 3/4 and so on. See if any of them will be accepted by your windows install.
Also, which windows (and is it 32bit or 64bit, home or pro) and how much ram altogether? Each windows version accepts only certain amount of total system ram. Make sure this is not what's limiting you.