Adaptive V-sync was created for that specific problem.
The only side effect of adaptive v-sync, compared to regular v-sync, is you'll get some tearing, instead of stuttering, when your FPS drop below your refresh rate. You'll have to try it to see if it is for you, but considering your problem, I expect you'll prefer adaptive v-sync.