There is nothing inherently different in the way that Windows handles a removable drive vs a non-removable drive (other than write caching). You can install regular apps on a flash drive, external hard drive, network drive, internal drive, ram drive, whatever. They will continue to work on the computer that installed them. The issue is portable apps, i.e., running on any computer you connect the drive to. That requires special applications that do not use the registry.
Drive letters don't change randomly, unless you assign that letter to another drive. And if it happens, you can change the drive letter back in 30 secs.