Actually it's not so much NTFS that requires you to use the "safely remove" icon, but rather whether the drive is configured with the "Quick Removal" option (this is set in the "Policies" tab of the device properties dialogue in the Device Manager app).
If the drive is set to "Quick Removal" then (in theory) you can remove it without worrying about doing a "safely remove". But if it's set to "Better Performance" then you're in danger of corrupting the drive if you just yank it out.
Either way, though, you should NEVER remove the drive if it's access light is blinking.
IMHO you're much better off getting into the habit of using the "Safely remove" icon. Otherwise it's probably just a matter of time before you screw the drive up somehow.
Remember that it's not just the file system you have to worry about, it's the device itself. Some of these high-performance flash drives use pretty large RAM write buffers that take a fair bit of time to empty - I've seen the activity light on my 32GB USB 3.0 flash drives keep blinking for 2 or 3 seconds after clicking "safely remove" and getting the "OK to remove" message. If you were to yank the drive out during that period it would likely cause problems.