Oblivion only uses 1 CPU core. Yep, one. Only one.
Although your i7 4770k is one of the strongest CPUs, mods can even bring it down while using a single core.
NPCs' radiant AI in Oblivion also takes a lot of CPU cycles, so it's common for fps to drop the lowest while there are NPCs on screen. They "solved" this in Skyrim by making NPCs do less and talk less.
If you installed a lot of scripted mods and NPC mods (IE, cats and rats, 28 days and a bit) that will also tax your CPU heavily since it's on a single core.
Find CPU monitoring software to test this. I'm almost certain you'll find that one of your cores is hovering between 90-100% constantly with the others at low load. If one of your cores is consistently higher than 90% load, then you've found the culprit, and you'll need to identify which mods or tasks are taxing your CPU so heavily.
Your VRAM is not the problem though; Oblivion was meant to run on 256MB and 512MB cards; it's virtually impossible even using every texture mod and lighting overhaul to push it over 1GB. Back when I had a 768MB GTX 460, I still ran a complete HD texture overhaul of Oblivion without running out of VRAM.
Remember, this is a game from 2006. Almost a decade ago. 3GB of VRAM is literally impossible to expend in Oblivion, even considering mods.