Tons of things can make a card unstable like that. A major cause being heat in a case. Believe it or not a lot of video cards are returned simply because of heat issues causing instability while a properly ventilated case would completely alleviate the problem. I know this from a shop owner locally who does a lot of sales of PC hardware and believe it or not, most units returned are in perfect functioning order.
However, there is always those few that do indeed have manufacturing defects/problems. A big thing here is the cooling devices not being properly applied with thermal paste/tape to allow appropriate heat transfer to the cards onboard cooling device. Also, bad VRM's, capacitors, and even the occasional DOA will happen.
So in your situation, here's what I recommend you do. First, make sure you have up to date drivers for the card you have. Then make sure you have appropriate case ventilation. A quick test method is to simply run the system with the case side completely off exposing the system to the outside air. I also recommend a small fan blowing air near the case (not directly in as it's not much benefit there) but near enough to keep any hot air from building around the system. If the problem persists, find the stock speed for the card and down clock it to the appropriate clock speed. If it still has these sorts of problems after making sure of good ventilation and a stock setting, it's time to RMA/Return the card as it is likely that there is a manufacturer defect.