"The best way to clean drivers is to first uninstall the drivers using the official uninstallers, reboot in Safe Mode and run Guru3D - Driver Sweeper. However, rebooting and not going into Safe Mode also works, but can cause some files not to be cleaned."
So the proper order (and correct me if I'm wrong) should be:
Uninstall drivers via official method
Reboot to safe mode, run driver sweeper
Turn off PC, uninstall video card, install new video card
Turn on PC, install new drivers