Whenever anyone tries to remove any media such as a usb flash drive, an external hard drive, etc. it forces you to eject the disk before unplugging. If you do not, It gives you a warning dialogue stating that memory could have been lost/damaged. However, in years of not ejecting before unplugging, I've never seen any memory corruption or loss of data. Why is it that it says this?
By default USB drives are marked as "Quick Removal", which means they don't cache write requests. So as long as you don't try to pull the drive out in the middle of copying or writing something to it you're not likely to cause any problems.
BUT - if you do happen to pull the drive out in the middle of a write operation, then the drive might end up corrupted. What the "safe removal" icon does is to block all subsequent writes to the drive and set it to "offline" status, so that you know beyond the shadow of a doubt that it's safe to pull it out.
It is always suggested to use safe removal to eject your USB Flash Drive and this is because it reduces the chances of unnecessary data loss from your flash drive and protects your flash drive from getting corrupt. You can also find more on USB drive data loss and data recovery here in this link:
I've never done safe removal for any of my drives. I've also never had an issue. As long as you aren't writing to the drive you are fine, the risk is that people in general don't pay attention to what they are doing to thedrive at the time of removal.
And jsut as a point of reference we're talking thousnads of unsafe removals, there really is very little to no risk if you pay attention to what you are doing.