If the GPU is of the same family, and it Unified Shader (made in the last 8 years) then no. It's been a very long time since you've had to.
A clean pull, then seating the other will work fine. I have been doing this for a while between a GT 240 and GTX560Ti on one system repeatedly.
I also have a backup ATi Radeon HD 6450 that works fine in the same machine without having to faff around with driver swapping.