Alcohol Prep Pads for site injections is what I normally use, they're 70% Isopropyl Alcohol and Sterile Water, they're the perfect size and very convenient.
If it's really bad you can use old school fingernail polish remover, which is Acetone and Distilled water, old school meaning it doesn't have a massive amount of additives, like Vitamin E, Fragrances, etc..
Use Q-tips to scrub tough areas with the Fingernail Polish Remover.
Finish with and Alcohol Prep Pad, and immediately dry with short spray of canned air which is normally co2, dries the alcohol instantly, and blows any loose fibers off.