Go ahead and use Afterburner with no fear. It will be compatible with the GTS250. Just start out with a moderate overclock, then keep adding until you start to experience problems (graphical glitches, lock-ups, or games that won't run), then back off the speed. You should be able to increase voltages, but it's hard to say what to shoot for. Again, start with moderate increases and see how high you can get. Afterburner also includes Kombustor, a stress test application that will allow you to test each setting for stability.
Unfortunately, I'm not seeing the GTS450 being all that great of an overclocker, with top clocks in the 770-800mhz range (without voltage tweaks).