It's probably just heat. Let the whole thing cool off, and reboot into safe mode. Without the overclocks set, if you still have problems like it glitching and artifacting when playing a game and/or running a benchmark, then you've probably broken it. Otherwise, just don't do it again.
It's actually kinda hard to break a GPU just by overclocking, especially when it's with afterburner or trixx or something. I've tried. Now if you wrote a custom bios and told it to do 2V and like 3ghz on the core, then it's a goner.