How are you uninstalling the drivers?
Are you using Add and remove programs?
That should work, but usually when going from AMD to Nvidia, you want to use a Driver Cleaner Program.
Haven't use one in years, so I can't recommend one.
The one I use to use was call "Driver Cleaner", but the name has been stolen so many times, I don't know which one is the original anymore.
I have tried it through device manager, through a driver cleaner program, and through add & remove programs. Every time I go to uninstall, my screen goes black to no return.