always increase the frequency only with small steps at one time during OC to fin the stable point.
If you made a jump by mistake, simply reset everything ASAP.
For OCing graphic card, I would first only start with gpu proc frequency, power limit, etc.+do not touch the VRAM freq yet until you maxed the gpu frequency, etc. while maintaining it stable.
If you have done this, not those settings down on a paper, just in case you would loose them. You can start fiddeling with the VRAM freq now.