Last things first; it would be the same for an external hard drive.
Most programs make entries in the windows registry, may rely on DLL files installed in /windows/system32 (or the 64-bit equivalent), and may even put .inf files in /windows. You can only do this with programs that are designed to be (or are accidentally) "portable," with all of their configuration and ancillary libraries installed in the one directory tree.
There are a gazillion sites (well, more than 10) dealing with this issue. For example, and only the first that came up with Google search, http://portableapps.com/ . In addition, some apps were put into portable packages as part of the BartPE project (Google it).
So. See if the program that you want has a portable version. For example, there's a portable version of FireFox so you can bring all your settings and bookmarks with you. Most Windows programs are _not_ portable, however.
Have fun.