Visit your profile settings, and click "Forum Options". There you'll see a link down at the bottom for "Personalized Smilies". Click it and you'll find a spot to upload your personalized individual smilies. Be aware though, that custom smilies go through an approval/review process by the administrators of the site (me and a few others) before being usable.
Most people use GIFs, though I'm not sure if it can take other types. JPEG and PNG might be possible, but for the size and typical complexity of a smiley (ie. virtually none), a GIF is probably the best. You are limited to 10kB after all.