Not entirely sure why the card cant support the factory overclock, I guess take it back to where you got it and get a replacement.
I have the Sapphire HD7870, and have overclocked fine.
Here's the settings if you want them (Using MSI Afterburner)
Core Voltage: 1237mV
Power limit: 0%
Core Clock: 1150Mhz
Memory Clock: 1375Mhz (equates to 5500Mhz in reality)
That is running stable for me with no noticeable artifacts or oddities. Though of course, the limits of overclocking is individual to every card.