You can overclock it BUT if it can't handle it then you will see junk happening on the screen. A while back I OC'd my 60hz monitor to 75hz and it was making parts of the screen randomly get stuff appearing/disappearing. I've heard that it can permanently damage the monitor if its done like I did it so I immediately stopped. (was just testing to see if it worked).
If your monitor says it can do 165hz then the monitor company probably has a special driver you gotta load so the Nvidia video card can detect 165hz.