Older GPUs used to consume system memory, not sure about newer ones. But basically that means communicating to the system memory over the bus, which is going to be a lot slower than the VRAM. Either the GPU will have to swap data in and out of VRAM to make up the gap or rely on the slower system memory for some of its data. You'll experience stuttering and FPS reductions when it happens. Ideally this is when you would lower the settings so the VRAM isn't exceeded.