Example:
I had Skyrim running on my HD5870 1GB card and it ran okay. I checked VRAM and it was pretty much maxed out though. I added the HD texture pack and performance dropped and I got a lot of stuttering.
I then upgraded to a GTX680. Yes, a much faster card but the point is that with the basic HD texture pack and no other mods I was going well over 1GB so it's probable that SOME of the stutter I had before was a lack of video memory (Skyrim still stutters for other reasons no matter what though).
Here's some basic numbers (Skyrim + HD Texture Pack + no mods):
VRAM (max) - 1.5GB
SYS RAM (max) - 1.75GB
SYS RAM is what the game ONLY used. Same for VRAM, though I can't test VRAM usage now with Windows 10 (in W7 I could disable advanced visual features and launch game with 30MB of VRAM usage).
Basically I need 2GB VRAM and 4GB System RAM to avoid major stutters. After that, tweak the game to get the optimal frame rate.
(Just one game example)