Just ran a few tests on a system much like yours (16GB ram, win 7 64 bit). And windows defender doesn't use much RAM at any time on my setup.
If you have antivirus software running, and then use defender they may conflict with each other (even MSE). My recommendation would be to disable it and run it on demand if you still want to use it from time to time.
On a side note if you want to investigate which program or DLL is causing the overload use ProcessExplorer from sysinternals. Its free and a great tool
Hope its of some use to you.