If your computer is stable and working then it is most likely fine, but electrostatic discharge (ESD), while not always instantly fatal to a sensitive component, can cause transistor damage that will shorten the life of a chip.
If you scraped or damaged any contacts on the CPU, yes it would stop working, but since your computer is still working. I think you are safe. And no, its not going to make your computer slower just touching the back of the CPU, although I guess you could compromise contact a little if you got an oily fingerprint on a contact, which would change your termination resistance, and cause some signal distortion... but its unlikely and probably won't matter anyways lol.
EDIT: Assuming bottom of CPU is the pin side.
If top side, you can get some fingerprint oil that can compromise a perfect thermal contact, but you're prolly OK.