Firstly I would recommend NOT enabling voltage tweaking for the time being. Make sure that in CCC, Overdrive tab, you first click the "lock" image so that it's "unlocked". Then, be sure that the Overdrive check box is Unchecked. After that, and with the Afterburner config file changed to allow unofficial overclocking, it should work just fine.
The problem with voltage tweaking is that not all cards support it. As emmerichem points out you could flash the BIOS to one that will have the voltage tweaking although to be honest I don't know for certain if that works, as I think I've also read it's a hardware thing. Also, it doesn't have to be the ASUS BIOS, it can be any card with a reference cooler on it, or the special Direct CU/Voltage tweak cards (those are the ASUS ones). For example I have reference model Sapphire cards and can voltage tweak.
BIOS flashing isn't really hard. I've done it many times on my 5850s using ATIWinFlash. Basically you open it, load the bios, and flick Program. However it does cause issues regularily when flashing (like screen blinking or the mouse freezes and unfreezes), but after a reboot it's always worked (and kept the flashed settings).
Without voltage control, it should be possible to get at around 850 core and 1150 memory. Voltage control will only help core OCing, as the memory voltage is separate and not really changeable (at least not on these cards)