You could use some compressed air or just use a paper towel to wipe the excess dust. Are you talking just on the outside of the card or inside it as well? If so, you can just remove the shroud and then wipe with a paper towel inside. What card do you have?
just the top part of the card, i don't dare clean the inside xD