Novuake :
You need a high GPU clock. You are doing it the wrong way around.
You can leave your memory clock at stock, the only reason people drop it is to conserve as much power as possible.
BTC mining does not use near the available memory speed that these cards have but can utilize all the available GPU power.
Repeat this until after adding a 5mhz step the hashrate goes down a LOT (like 50-100 kh/s drop). You should be able to get to 600+ kh/s this way. Once you get to your "peak" engine speed - where going past it in engine clock actually noticably slows down your hashrate - take note of the RATIO of core clock vs your memory clock (1500) - it'll usually be around 0.61. (ie: memory is 1500 mhz, gpu is 915mhz: 915/1600 = .61).
Armed with your "magic ratio" (this is different for every card)... start clocking up the memory in 10mhz steps until your system hangs (leave the engine core at whatever speed it was at before, don't touch it just yet even if your hashrate goes down).
After it crashes, take 20 off of the memory speed you achieved before crashing.
Now, take that final memory speed, and multiply it by your magic number (.61 for example) and that will be around the engine speed you want to ideally be at for best hashing, check +/- 20 mhz from this number in increments and see which yields your best speed.
For example, if you got to 1650mhz before it crashed, back down to 1630, and put your engine/core up to 994mhz (1630 * .61). This should give you a good hashrate ~> 700kh/s. If it's LOWER, then lower your gpu speed by a few mhz until it springs up to a nice hashrate. This is your sweet spot. Put this memclock + gpuengine in your cgminer startup script."
Is what I went off of.
I'll try your method and see if that works!