Improperly removing a USB drive is dangerous because data may still be waiting in a cache. You may see a "delayed write failure".
Have a look in Task Manager and try to determine which task "owns" the drive.
You could also configure the drive for "quick removal". I believe this effectively disables the write cache.