What manufacturer is the card from? Gigabyte recently had a defective batch that had this problem, I bought a gtx 560 and the drivers kept on crashing. The card did this on other systems too. I returned it and bought an Asus and everything is better now. The problem seemed to be in the card itself, if it was the PSU's fault, your whole system would restart.
I have the MSI Twin Frozr III PE, and true about the PSU... I'm just confused at this point. I haven't heard of this happening before so I'm not entirely sure where to even begin aside from trying different driver versions.