It doesn't matter how much VRAM a gae can use because the settings that you'd need to enable in order to use so much VRAM would already be too much for the GPU to handle regardless of the VRAM being 4GB or 8GB.
When something has to load over the PCIe bus, yes, that causes huge performance drops (as far as single digit FPS). However, you would never encounter that on a 4GB GTX 980M in gaming regardless of the game. The GPU doesn't have enough performance for the settings that would need more than 4GB anyway.
Also, one thing to realize is that a game will pretty much fill up the VRAM even when it doesn't need to. Basically, jsut because a game may use say 4GB doesn't mean it actually needs that much. In most cases, 2GB is still enough despite what game companies want to say. 4GB is not an issue at all for the GTX 980M unless you have two of them in SLI and even then, only a few situations would struggle and again, if you're looking for that kind of performance, a desktop makes more sense.