A toothbrush might be abrasive and scratch the contacts. I use compressed air for cleaning out ports. If it still works, there's nothing to worry about.
As soon as I saw the title that's the first thing I thought of. It will probably be ok but next time I would just use compressed air. If you really need to clean it out you can try a q-tip moistened with rubbing alcohol but as said make sure the computer is off and UNPLUGGED. Also make sure it is compleatly dry before you go sticking a USB device in there.