strangestranger, xaria - Thanks ! It worked like a charm.
So I logged into the windows safe mode. I've a dual boot with Linux( Ubuntu ) and Windoze and so it was tricky. After couple of trials I was able to get to the windoze menu ( press F8 at regular intervals ). Once I was in safe mode, there indeed was a removal utility from ATI itself ( ATI being my old card ). I restarted, came back to safe mode, shutdown and installed the new (nvidia) card.
Thanks for the help.