It's not damaged, don't be afraid. Imagine what happens to a car when you pull out its driver while he's driving it?
The same happened - you uninstalled the drivers windows didn't switch to the proper generic drivers (still they switched, otherwise it would have been black screen) and that's why the display was buggy. When you switched the port - windows managed to detect right drivers apparently. When you reinstalled the real driver, everything came back to normal