Overclocking causes the GPU to make errors, it's called 'artifacting'. It just means you OC too far. Start with little increments and work up, test under load at each step. Heat is the main factor, anything you can do to cool the GPU will help, blowing out the fan and heatsink for example. GPUs will only overclock a marginal amount so don't expect too much. You could stand a video card upgrade at this point honestly.