Use a bit of alcohol on a lint free cloth and wipe it off with that, your worry about the oil from your hand is getting a bit anal. If your worried about that wipe your finger with the alcohol first. There will not be a whole lot of difference between the silicon grease and the silver based stuff if both are applied right in the first place so it is kind of chasing your tail.
Mike
I know one thing for sure, I don't know [-peep-]!