When using a USB memory pen, is it necessary to use the windows "safely remove hardware" option or can you just yank it out without fear of anything bad happening???

Thanks

when you are dealing with memory, use the safe way.

It depends on whether you have enabled or disabled caching to the drive. Go to the device manager and assuming you have the drive plugged in, select it under disk drives and double-click on it. Then go to the policies tab in the newly opened window. If "optimize for quick removal" is checked, then you can just pull it out, if "optimize for performance" is checked, you need to use the safely remove hardware thing.

EDIT: I always just pull it out and have never lost data.

The "safely remove hardware" bit is used because removable drives like USB sticks and camera memory have a file system on them just like your hard drive does. Most of these drives are small (<= 32 GB) so they use FAT16 as the file system type. FAT16 is an older file system and writes to the disk are simply done on demand. If the power to the machine is cut or the drive is simply yanked out during a write, this will result in incomplete data and possibly a corrupted file system on the drive. The "Safely Remove Hardware" bit calls for all programs accessing the drive to complete whatever reads/writes they need to as the drive will be removed. When all I/O is complete, the OS removes the drive from its list of usable drives and then pops up the bubble telling you it's safe to remove it.

If you formatted your memory card/USB stick yourself and put a journaled file system like NTFS, ext3, XFS, ZFS, HFS+ or ReiserFS, then the removal is different. All writes to the drive are first written to a journal, which is a temporary space on the drive. Once the data is written to the journal, then the data in the journal is written to its final location on the drive, overwriting the original data. Yanking out the drive will cause the following things to happen:

1. Yanking the drive out before the data has been completely written to the journal causes the original data on the drive to remain safe. The changed version of your file that was being written to the journal is lost as the FS sees that there is an incomplete write to the journal and flushes the journal the next time the drive is accessed.

2. Yanking the drive out after the data is written to the journal but before it can be completely written to the file system results in the partially-modified file in the file system being replaced with the good copy on the journal the next time the device is accessed. The user would never know that the final write never completed.

3. Yanking the drive out before any writes to the journal occur or after the data from the journal was successfully copied to the file system cause no issues as the drive is idle.

Safely removing a disk with a journaled file system causes the same actions as with the non-journaled file system but will also cause the disk to make sure that all of the journaled data has been written to disk successfully. Journaled file systems are more modern and much more reliable than non-journaled file systems. They still should be removed safely, but the consequences for not doing so are basically that it takes a longer time to access the disk upon the next access (due to journal-file comparison and flushes.)

The reason that journaled file systems do not commonly appear on UBS sticks or camera cards is because of Microsoft. Most people run Windows computers and Windows computers can only recognize FAT16/FAT32 and NTFS file systems out of the box. ext3 support can be easily added via a driver, but the other file systems require some external program to access (ReiserFS) or are completely unreadable in Windows (XFS, ZFS, possibly HFS+ as well.) NTFS isn't used on removable devices because it is covered by Microsoft patents while the FAT file systems are direct descendants of the original DOS file systems and are in the public domain. Device makers would have to pay MS a royalty fee for every device shipped if it was pre-formatted with NTFS. So the device makers have a few options:

1. Ship the drive with FAT formatting for free even though it's sub-optimal.

2. Ship the drive with NTFS formatting and pay MS $0.25 (IIRC) per unit shipped.

3. Ship the drive unformatted for free and require the user to format it on their computer, where they can format NTFS for free. This axes out-of-the-box working ability.

4. Ship the drive with ext3 for free and require that the users download the Windows ext3 IFS driver for free. This also axes out-of-the-box working ability and renders the device unusable on computers where the user doesn't have admin access to install the ext3 driver (work computers, public terminals.)

Most manufacturers want easy-to-use and cheap rather than good so they ship FAT-formatted drives. I leave mine formatted as FAT as cameras won't support anything but FAT16 and since I don't run Windows, I can r/w to FAT but basically only read from NTFS (write support is VERY limited.) I'd simply format my USB stick ext3 and be done with it, but I use my drive a lot on university computers and I highly doubt that the techs would install the ext3 drivers for me.

This is a great topic and info.

I have an OCZ 2GB Rally USB drive and the stinking thing will never allow itself to be "safely removed." IIRC it did allow this process the first or first few times I used it but in the 100+ times since it is a no go. I just yank the thing and, though I have never lost/corrupted any data to date, I worry that I will. Any thoughts? I can't say for sure but I certainly don't remember reformatting the drive.

I am going to check out cache issue randomizer mentioned next time I use it. Perhaps that is the issue.

.

Great post MU_Engineer.

I have a 4GB flash drive I use just for readyboost on Vista, can I format it NTFS or ext3? Would there be any performance gains by doing so? What is the benefit of being able to use ext3 in Windows (does the driver work with 64bit Vista?), would it allow me to use my ext3 formatted drives in Windows? (I have triple boot XP, Vista, Ubuntu 7.04).

cutthroat,

The benefits of being able to use ext3 in Windows mostly stems from the fact that all Linux machines have support for ext3. If you wish to dual-boot a Windows and Linux machine and have a shared My Documents/home directory, this had to be done in FAT16/32 in the past as that was the only FS that both could read reliably. Windows doesn't like FAT32 >= 32 GB much, so this limited the shared partition. ext3 is a much more robust FS than FAT16/FAT32 and would be much better for a shared partition on a dual-boot system. Passing an ext3 USB stick between suitably equipped Linux and Windows machines would be much better than one formatted with FAT.

There is no support for the ext2/ext3 IFS driver in any x86_64 Windows operating system and no official support in Vista yet. Perhaps you could use the 32-bit XP driver in 32-bit Vista, but I'd not know if that would work. The driver supports Windows Server 2003, so since Vista's kernel is based off of WS2003, perhaps it would work with the 32-bit version...maybe. Here is the web site with the ext2/ext3 Windows driver.

So how would you format a flash drive into NTFS then? In windows XP (32 bit) I can only format it into FAT or FAT32. For XP 32 bit would there be any reason to have it formatted into NTFS other than the reasons regarding removal of the drive?

Hmm. Maybe Windows won't let you format the drive as NTFS if it is under some certain size. The advantages of NTFS over FAT16/FAT32 is that it is generally more robust, regardless of the device is uncleanly removed or not. NTFS also supports UUIDs and ACLs, which could possibly lock you out of your own files :?

no, eventually you'll get corrupt data on the usb drive

See this article for file size limitations, defragmentation on FAT, etc.:

http://en.wikipedia.org/wiki/File_Allocation_Table

and this for ext3:

http://en.wikipedia.org/wiki/Ext3

Q1. What are the (potential) consequences of a corrupted file system on the drive - will you "just" lose all data on the drive, or will the drive become permanently unusable?

Q2. For a standard windows user, using a drive that has been formatted to NTFS, if the drive is removed 'unsafely': I understand that the changed version of the file that was being written to the journal can be lost, but by using NTFS do you completely prevent any potential file system corruption?

Thanks in advance.