ive had this happen many time before because i wasnt careful. brute force doesnt play well with electronics most of the time
next time try heating the heatsink up with a blow dryer, then carefully twisting the heatsink left-right while GENTLY pulling up. it'll come off.
and yes, you can bend/break the contact points off