You will drop back to single channel with your proposed solution. Not really a big deal.
Also, if your BIOS memory setting is set to AUTO, the system will default (properly) to the slower of the installed memory modules.
Your best bet would be to install matching 4GB modules. But the mixed modules should work, only at the slower speed and in single channel (really not a big deal in and of itself).