Sign in with
Sign up | Sign in
Your question
Solved

Help please. Does anyone know how HD formatting works?

Last response: in Storage
Share
June 6, 2010 6:33:01 PM

I just read the wiki page about HD formatting but it raised more questions than answers.
I need to understand more about the Reinitialization as defined in the wiki page and especially the feature that can test each sector of the hard drive and marks the bad sectors so they are not used and thus prevents hard disk errors.

1. Using Disk Management in Windows, I can format a hard disk with Quick Format option unchecked. Is this Reinitialization that marks the bad sectors? If not is there and can you please, suggest a program that can do that?

2. How does marking of the bad sectors with Reinitialization work?
My guess is that the formatting software will write information on some part of the hard drive about where the bad sectors are and other software will use this information to skip writing on these sectors. Is my understanding correct?

3. If my understanding is correct, what actions can remove the information about where the bad sectors are. For example will this information still stay after doing things like setting hard drives as RAID, deleting or creating volumes and partitions, and reformatting using the Quick Format feature?. So said in other words, my question is, do I need to reformat disks every time after such actions?

I will greatly appreciate your input.

More about : formatting works

Best solution

a c 328 G Storage
June 11, 2010 3:50:26 AM
Share

To start, as the wiki page says, no normal user can or should try to do a real "Low-Level Format" - that is done ONLY at the HDD factory these days. However, too many people still use the term for something else.

The wiki article fails to clarify the difference between a Quick Format and a Full Format done by Windows. I'll get to that; suffice to say for now the Full Format is the process that finds faulty sectors and marks them so they don't get used. But you expressed an interest in permanently removing "bad sector" markings.

Modern hard drives actually have their own low-level controller on the card that is part of the drive. It has its own processor, BIOS of sorts, and non-volatile memory as well as volatile RAM used for its cache. When the disk is first made and Low-Level Formatted, all of its data tracks are created with markers on them, etc, and they are tested for performance. There are always too many good sectors for the design size of the unit. So the required good sectors are made available, and the good spares are noted in a list of spares in the HDD's non-volatile memory. Windows knows virtually none of this - all it knows is the drive says it has a certain number of sectors ready, and they're all good.

On a brand new empty HDD, there are two tasks required before Windows (or any OS) can use it. First all the unorganized space has to be assigned to usable blocks called Partitions. A Partition is one contiguous block - that is, there are no gaps in it - that Windows will identify and use as a "disk". You might create only one Partition on an HDD that consumes all of the space it has; you might make one smaller than that and leave the rest of the space Unallocated until later; or you might make two or more Partitions, and each will be treated by Windows as a separate "disk" with its own letter name. When you make the first Partition the process writes to the start of the HDD a Partition Table containing details of where the Partition starts and how big it is. If more Partitions are created later, their specs are added to this master table. That way any software can find the details of disk space allocation from this place on the disk. The exact format of data in a Partition Table is highly standardized so many OS's can use them.

Once a Partition has been created, the second step is to Format this Partition. That means basically writing to the start of that particular Partition a set of files used to hold the Directory data and the table of sector allocations, etc. This creates the File System, because the details of what information is written in these files, and how, is the basis for the File System installed by the Format process. The most common system used in Windows now is NTFS, but you still find some FAT32 File Systems on some HDD's. File systems are all different, so to use a formatted disk the application or OS must understand that particular File System.

In doing the Format operation, Windows offers two options. The Quick Format option creates those tracking files and writes zeroes to them so they are "blank" and ready to use, but that's basically all it does. Even later if you do a Quick Format again on the same drive, it will do that same job, which results in a blank set of file tracking files at the front, but NO changes to the actual sectors that contain data.

The other option is a Full Format. This does a Quick Format first. Then it proceeds to test absolutely every sector of the disk (that is, this Partition only) to see if it contains good readable data. Even on a new empty drive, all sectors do contain readable data placed there by the factory processes - mostly just zeroes. If any sector fails to read properly, Windows goes to its own sector allocation table and writes for that one sector a special code which says "This sector is no good - never try to use it". From then on no application or Windows will use it. The interesting question I cannot answer is this: given a HDD that has been used, if one does a new Quick Format, does that preserve the old notes about bad sectors in the new allocation table? Or, is that info lost?

Some of the newest versions of Windows add a step in the Full Format. First they fill the sectors with zeroes, THEN they read them back to test the sector. In this way they actually do destroy any old data with a Full Format, whereas previous Windows versions did not.

For purposes of Creating Partitions and Deleting them and to Format a Partition there are tools built into Windows. The Format and CHKDSK tools are available in many places. The Creation and Deletion of Partitions can be done in a tool called Disk Management. But to make starting to use a new disk easy, many disk manufacturers provide a utility to "prepare" the new disk or "initialize" it by doing these tasks for you. Note that this use of "initialize" is not what the Wiki article talked about. That article seems to be describing the Format operations, except that they also talk about having the zero-fill operation done, too.

Separately from the Format tool, Windows has a tool called CHKDSK. It will run completely through a disk, to do a non-destructive test of each sector. It's like the older type of Full Format in that it (almost) never writes to the sectors, it just reads them and looks for data that has errors. (However, CHKDSK does NOT empty out the Directory and sector tracking files!) If it finds any "bad sector" it marks that sector as bad, then adjusts the file tracking records to fill in that gap with a new unused sector from somewhere else. (This is the one operation that alters the disk contents in CHKDSK.) The trouble here is that the resulting file now has a sector's worth of foreign data stuck into its middle, so the file is corrupted. It may be useless and need to be deleted and replaced. But at least the rest of the disk is OK, and the bad sector will not be re-used. CHKDSK also does a number of other tests for file integrity, too.

Both the Full Format and CHKDSK routines have to work their way through EVERY sector on the disk. On today's units of 1 to 2 TB, that usually takes 4 to 7 hours! So don't wait around and watch!

So you see, using Window's tools you can Create and Delete Partitions out of blank space on a HDD, you can Format a "disk" (Partition), either Quick or Full, and you can re-do a Format of either type. Doing a Quick Format on a used disk removes all the file tracking information, but technically it does not damage the data in the sectors. It just makes it really hard to figure out which data belongs to which old file!! A Full Format will cause Windows to make notes about any faulty sectors in its own sector allocation tables on the "disk" and substitute other sectors for them, but it cannot preserve any of the data in the bad sector.

Now, completely out of Windows' view, the hard drive itself is doing a bunch of other work at the same time through its own on-board processor. Every time Windows uses the disk for anything, it's actually the HDD unit's processor that does the job on behalf of Windows, and it does it own things at the same time in secret.

Every time it reads a sector, it does some measurements on the quality of the electronic signal coming off the disk heads. If it finds the signal is weak it can decide that this is a predictor that the sector will actually get worse and become bad soon. So before that happens it tries (and usually succeeds) to read good data from the sector. (It actually uses more robust processes to read weak data signals than Windows can use, which is one reason it succeeds.) It then writes that data to one of the known-good sectors it has in its "spare" list and substitutes that for the poor sector. Then it retires the poor sector to never-use-again land. If all of this works properly the data is saved before it gets too bad to read and Windows will never know this all happened! BUT the HDD's board keeps track. One of the things it can do if this process uses up a LOT of the spare good sectors is to throw out a warning message via the SMART monitoring system. The message basically says, the HDD is running short of good spares, so you better plan to replace it soon while it can still keep up the process.

Two things can still go wrong with this hidden system. One is that some catastrophic event, like dropping the HDD, might cause severe damage to one or more sectors so that the data in them really is not just weak, it is truly unreadable. In that case although the error will be found and the sector substitution made, the old data will be lost. This is very similar to what Windows does when it finds "bad sectors". The difference is, that is the ONLY thing Windows can do, whereas the HDD's background system can do better things most times. The other thing that can go wrong is that a sector does get weak but it never is actually called for, so the HDD's processor board never sees that problem - it goes on undetected and uncorrected, maybe until too late to preserve its data.

Now, many current HDD's can be forced to use their hidden processes for certain operations. To do these things you usually need to download and use some disk diagnostic testing utilities from the maker of the HDD unit. One tool is called a Zero Fill - that old phrase. THIS IS TOTALLY DESTRUCTIVE FOR DATA. So you MUST preserve your data before doing this!! What it does, though, is force the HDD's on-board processor to go through the whole disk and write zeroes to every sector, then read them back to check for valid data. In doing this it WILL get to test EVERY sector and hence it will find ANY poor-quality sector and replace it from its spares list. When it's done the HDD is truly empty (well, full of zeroes) like a brand new unit and must be Partitioned and Formatted again. But if you run a Full Format on it, Windows will find there are NO "bad sectors" to mark off. Then you can restore old data to the rejuvenated drive. That's a lot of time and work to do, but it does make the HDD appear to Windows' view as a perfect new unit.

Creating a RAID array actually does a custom version of creating a Partition Table on each of the HDD's involved in the array. After that's done, normally you will end up Formatting the "drive" in Windows, and I'm sure this process will NOT preserve any old records of "bad sectors". In fact, there might even be some "bad sectors" if the HDD's on-board background process has not detected them yet and corrected them, but if you do only a Quick Format, Windows won't find them and mark them off.

To answer your last question, yes. Every time you Create a Partition or Create a RAID array, you will have to Format it. If you run a Format (either Quick or Full) on a used drive, that will re-establish a blank set of tracking files at the start of that "drive" (Partition), and a Full Format also will check for and mark off any "bad sectors" and MAY zero-fill the drive, too. But unless it does that zero-fill operation it does NOT destroy old data, although it takes a LOT of sophisticated work sometimes to recover data from a Formatted drive. (Actually, you can buy pretty sophisticated Partition Recovery and Data Recovery programs to try to recover from disasters.) No Format operation will make any change to the way that HDD space is allocated to Partitions, though.
June 11, 2010 2:00:47 PM

Thanks "Paperdoc" it's a great post. I found these information really helpful and I am keeping a copy of this post please don't mind it. I were looking for such lucrative information on HDD and at last I found so why I will not keep it.
Related resources
a c 328 G Storage
June 11, 2010 3:00:59 PM

You're welcome. That's why these forums (fora?) exist - so people can help each other. If you are saving somewhere, be aware I just made some minor changes this morning - June 11/10.
June 11, 2010 4:40:38 PM

I just feel the need to say, that was truly fascinating. Before reading your post, I had only the vaguest idea of how hard drives did this kind of "self-repair" process. Now I feel like I understand it 100% better. Awesome explanation.
June 18, 2010 3:47:28 AM

I almost missed this - I thought no one is replying.
What a great post Paperdoc! Unlike most of the information about this out there, you put it simple and clear. Very much appreciated.

I often restore hard drives using disk images. I use Norton Ghost to make a disk image of my system drive after fresh installation of the OS and programs. Later I make restorals in case something goes wrong or after changing my hard drives and RAID configurations. For restoral Ghost requires unallocated disk space because the disk image contains all data about the partitioning, formatting, and apparently the bad sectors if any.
Now from the new things I learned, I gather that formatting a hard drive intended for disk image restoral is a waste of time. Also restoring to a different hard drive than the original makes the information about the bad sectors in the disk image invalid. I guess in such situations Check Disk should be run to ensure that Windows will have the correct information about the current bad sectors. Hopefully Windows is smart enough to replace such information instead of adding to it.
June 18, 2010 3:57:54 AM

Best answer selected by Startup123.
June 18, 2010 1:23:53 PM

very nice!
a c 328 G Storage
June 18, 2010 6:26:40 PM

Startup123, in regard to your comments on Ghosting and format, etc.:

If I understand the Ghosting process correctly, you are quite right. It will EXACTLY copy the info from every sector of the Source Disk to the Destination. Within that information are the contents of the sector allocation files that the File System maintains. It is in there that special characters are placed to indicate that a particular sector number is "bad" and should never be used. So if that info is copied to a new HDD unit, the new unit's files now falsely show that particular sector number to be "bad", and Windows will NEVER use it. At least, that is, until that HDD is completely re-formatted with a Full Format that re-does all the sector tests and rebuilds the information of "bad sectors". But of course, you would not be doing that on a disk image full of data you are preserving.

BUT there is a missing item here that needs fixing. The Destination HDD now has a complete copy of all the Source HDD's data including its notes on "bad sectors". What it does NOT have, though, is any record that the Destination HDD itself may have other "bad sectors", because the HDD area that contains that data (the sector allocation tables) have been re-written in the Ghosting process! In fact, it's even possible that good data from the source has been stored in faulty sectors on the Destination HDD with nobody aware of this problem. The solution surely is to run CHKDSK on the Destination drive after it has been written so that any "bad sectors" that actually do exist can be detected and noted properly by updating the allocation table. This process, however, has two potential flaws. One is that if there really are good data stored in "bad sectors", there is no data recovery process involved in what CHKDSK does. So, although the probability is VERY small, you just MIGHT have a corrupted file on the Destination HDD that was just written and checked. The other is that CHKDSK does not make any attempt to test and "re-qualify" (for lack of the proper phrase) a truly good sector of the Destination HDD that has been falsely marked as "bad" by the Ghosting process. So after CHKDSK has been run it is possible that a VERY FEW sectors will be OK but marked as bad and never used. That really is NOT a problem in most cases. Unless, of course, the original source Disk had HUGE numbers of "bad sectors" marked off by Windows before the process started.

It is possible that the HDD's own hidden background processes might prevent much of this - I'm not sure. Consider this way: under the Ghost process, as that application under Windows writes data to the destination HDD, that piece of hardware's background process should actually be checking every disk write operation for good data in the sectors it has just written. This will be especially true if the Ghosting software actually reads back what it has written to verify that it is done properly. In that case, the HDD's processes will detect any truly questionable sectors and replace them automatically, ensuring the everything written freshly to this HDD is on good sectors only, and Windows will never know that the background checks and fixes are being done. In this way the possibility that data are written to bad undetected sectors of the Destination HDD is actually eliminated, and there should NOT be any danger of data corruption.
!