It's a common problem & doesn't indicate anything wrong with your external drive.
By default, the "removal policy" for external drives is set to "Quick Removal", which means you don't have to use the "Safely Remove Hardware" utility at all, even though the drive in question is listed there.
Just ensure that the drive isn't reading or writing (check it's activity lamp) and you can safely unplug it.
I never use the "Safely Remove Hardware" now for any of my four external hard drives and pen drives.
I just ensure they are not busy then unplug them. It's never done them any harm at all.
Of course, if you change the removal policy to "Better Performance" (which tells Windows to use delayed-write caching) you must use the "Safely Remove Hardware" utility or just leave the drive connected till you shut down your computer.
Want to know where to find or change the current "Removal Policy" for a particular drive? It's in Device Manger. Double-click the drive you want to check, then click the "Policies" tab.