This is how I do it and it works every time without issues.
1: Download Drivers and leave them on the desktop
2: Uninstall drivers from the control panel
3: Run DDU to wipe drivers and reg files completely and tell it to not restart but to shutdown.
4: Pull old card, install new card
5: Turn system on and use drivers on the desktop.
People skip steps like just using DDU or jut uninstalling drivers tend to run into problems. More so when switching brands but I've seen the same problem just switching series from NVIDIA and AMD.