A 9600GT should be able to offload AVC if the video and player are DXVA compliant.
Are you running XP? VLC added DXVA video acceleration, but it only added DXVA v2.0 which requires Vista, 2008, or 7 to use. XP uses legacy DXVA 1.0.
Also, what is your CPU usage when playing the MKV?