Well if it runs normally without the mods then it is obvious that one of the mods is causing the problem and not much you can do about it. Have you tried Bethesdas own HD texture pack from Steam instead of a mod??
I get about 1.4gb vram usage on both of my 7970s at 1080p so i would think that your vram would be fine unless you are playing at 1600p? or multi-monitor?