Leave your mem slider at stock speed and incrementally increase your clock speed. Once that fails, use the last setting that worked. Once you find your max core clock speed, then play with your mem clock.
Is there a reason you don't want to up the voltage? You would see much more OC potential if you were willing to bump the voltage a bit