Update 1:
An update on the situation: With ImageMagick and a Python script to parse its output, I discovered 200+ invalid image files in the 30GB image directory recovered by ZAR 9.2. (So seemingly ZAR is far more reliable over GetDataBack in this case.) Around 50 of them are successfully recovered with GetDataBack NTFS, and 16 ones are only present in the GetDataBack copy. So I have 177 corrupted images right now, around 0.2% of all recovered files. ImageMagick probably is not capable to discover all image files with issues, though, and the thing I'm still worrying is, are there still missing files neither programs are able to recover?
I also manually verified some videos files. Looks all good, but I found some files missing.
Restorer Ultimate discovered the "half" directory tree as well, but presently I don't have enough space to let it save the recovered files...
The environment
---------------
Windows 7 x64 Ultimate SP1, Gentoo ~amd64, Hackintosh 10.9.4, FreeBSD 10
Seagate ST1000DM003 (1TB)
The problem
-----------
The MFT of one of my 340 GB NTFS filesystem was heavily damaged by chkdsk, after an improper unmount (under Gentoo, using ntfs-3g). (Not my system partition, so it's still safe to install stuffs in Windows.) Indices of all entries in MFT are incorrectly removed, resulting in an almost empty filesystem tree.
Here's "\System Volume Information\Chkdsk\Chkdsk20141006090640.log", showing how the MFT was messed up by chkdsk:
The data I wish to recover is around 70GB in size. One directory occupying 30GB with somewhere around 100,000 images of varying size (5KB - 16MB, usually several hundred KB in size), and two others with 40GB of video files of varying size (up to 3GB).
The filesystem is probably moderately fragmented since it's almost full and files are frequently added/removed.
I have not modified anything in the partition, but I can't guarantee Windows and the applications in background didn't do anything.
I tried recovering data with GetDataBack NTFS 4.32 yesterday, in "systematic file system damage" mode. ("Undelete" mode doesn't display the lost files, by the way.) 7 filesystems are found, 3 of them are broken, 4 of them work in a certain degree. I'm able to get half of the filesystem tree (some directories and files are missing), with correct metadata, however after extraction I found many files got apparently corrupted contents (missing JPEG SOI marker, missing FLV header, invalid video frames, etc.).
I'm playing with ZAR (Zero Assumption Recovery) 9.2 right now. Got basically the same "half" directory tree as GetDataBack. ZAR's validation detected thousands of invalid files (I believe by validating file signature). But seemingly at least the number of broken files is probably lower than the amount I got from GetDataBack.
I have not discovered clear signs of bad sectors or hardware failure on the hard drive so far, and the hard drive is about 1.5 yr in age. I don't think there's any damage to the GPT partition table, either.
The most recent backup are from like two years ago (on my old computer)... And I don't have the hashes of files, either, so I could only validate the files by viewing them manually or with the file signatures.
Any advices? Thanks in advance.
By the way, I guess I will switch to ext4 for storing those files in the future... The accident gave me a terrible on the reliability of NTFS -- why would a simple power loss kill a filesystem?
An update on the situation: With ImageMagick and a Python script to parse its output, I discovered 200+ invalid image files in the 30GB image directory recovered by ZAR 9.2. (So seemingly ZAR is far more reliable over GetDataBack in this case.) Around 50 of them are successfully recovered with GetDataBack NTFS, and 16 ones are only present in the GetDataBack copy. So I have 177 corrupted images right now, around 0.2% of all recovered files. ImageMagick probably is not capable to discover all image files with issues, though, and the thing I'm still worrying is, are there still missing files neither programs are able to recover?
I also manually verified some videos files. Looks all good, but I found some files missing.
Restorer Ultimate discovered the "half" directory tree as well, but presently I don't have enough space to let it save the recovered files...
The environment
---------------
Windows 7 x64 Ultimate SP1, Gentoo ~amd64, Hackintosh 10.9.4, FreeBSD 10
Seagate ST1000DM003 (1TB)
The problem
-----------
The MFT of one of my 340 GB NTFS filesystem was heavily damaged by chkdsk, after an improper unmount (under Gentoo, using ntfs-3g). (Not my system partition, so it's still safe to install stuffs in Windows.) Indices of all entries in MFT are incorrectly removed, resulting in an almost empty filesystem tree.
Here's "\System Volume Information\Chkdsk\Chkdsk20141006090640.log", showing how the MFT was messed up by chkdsk:
------------------------------------------------------------------------------
Checking file system on E:
The type of the file system is NTFS.
Volume label is Backup.
One of your disks needs to be checked for consistency. You
may cancel the disk check, but it is strongly recommended
that you continue.
Windows will now check the disk.
Unable to query LCN from VCN 0x9 for attribute of type 0x80.
The non resident attribute of type 0x80 is inconsistent. The valid data
length is 0x6a40000, file size 0x6a40000, and allocated length 0x4000.
The non resident attribute of type 0x80 is inconsistent. The valid data
length is 0x6a40000, file size 0x4000, and allocated length 0x4000.
CHKDSK is verifying files (stage 1 of 3)...
Deleted corrupt attribute list entry
with type code 48 in file 0.
Unable to find child frs 0x24 with sequence number 0x5.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to find child frs 0x26 with sequence number 0x2.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to find child frs 0x25 with sequence number 0x1.
The attribute of type 0x80 and instance tag 0xa in file 0x0
has allocated length of 0x6a40000 instead of 0x4000.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to locate attribute with instance tag 0xa and segment
reference 0x1000000000000. The expected attribute type is 0x80.
Deleting corrupt attribute record (128, "")
from file record segment 0.
16 file records processed.
File verification completed.
0 large file records processed.
0 bad file records processed.
0 EA records processed.
Correcting file name errors in system file record segment 0.
0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
An index entry of index $I30 in file 0x5 points to file 0xcc1
which is beyond the MFT.
Deleting index entry $RECYCLE.BIN in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd8f1
which is beyond the MFT.
Deleting index entry .DS_Store in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xf06
which is beyond the MFT.
Deleting index entry .Trashes in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1784c
which is beyond the MFT.
Deleting index entry .windows-serial in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17328
which is beyond the MFT.
Deleting index entry 3DReaperDX in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4eb
which is beyond the MFT.
Deleting index entry bittorrent in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x103dd
which is beyond the MFT.
Deleting index entry books in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4e9
which is beyond the MFT.
Deleting index entry Downloads in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4e9
which is beyond the MFT.
Deleting index entry DOWNLO~1 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x11
which is beyond the MFT.
Deleting index entry found.000 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x12337
which is beyond the MFT.
Deleting index entry found.001 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x10
which is beyond the MFT.
Deleting index entry Games in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xf10
which is beyond the MFT.
Deleting index entry img in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xfe4a
which is beyond the MFT.
Deleting index entry log2.txt in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x118a0
which is beyond the MFT.
Deleting index entry mom in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1779e
which is beyond the MFT.
Deleting index entry photos in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17746
which is beyond the MFT.
Deleting index entry shufa in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17753
which is beyond the MFT.
Deleting index entry substance-designer-tut in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbd
which is beyond the MFT.
Deleting index entry System Volume Information in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbd
which is beyond the MFT.
Deleting index entry SYSTEM~1 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x23
which is beyond the MFT.
Deleting index entry TheSims3 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbf
which is beyond the MFT.
Deleting index entry tmp in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17772
which is beyond the MFT.
Deleting index entry tools in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x11cd7
which is beyond the MFT.
Deleting index entry universities.txt in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xc37e
which is beyond the MFT.
Deleting index entry videos in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xdb6c
which is beyond the MFT.
Deleting index entry videos2 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1731b
which is beyond the MFT.
Deleting index entry work-projects in index $I30 of file 5.
Index entry $ObjId of index $I30 in file 0xb points to unused file 0x19.
Deleting index entry $ObjId in index $I30 of file 11.
Index entry $Quota of index $I30 in file 0xb points to unused file 0x18.
Deleting index entry $Quota in index $I30 of file 11.
22 index entries processed.
Index entry $Reparse of index $I30 in file 0xb points to unused file 0x1a.
Deleting index entry $Reparse in index $I30 of file 11.
Index entry $RmMetadata of index $I30 in file 0xb points to unused file 0x1b.
Deleting index entry $RmMetadata in index $I30 of file 11.
Index entry $UsnJrnl of index $I30 in file 0xb points to unused file 0xcbc.
Deleting index entry $UsnJrnl in index $I30 of file 11.
Index verification completed.
0 unindexed files scanned.
0 unindexed files recovered.
Creating object id file.
Inserting an index entry into index $I30 of file 11.
Creating index $O for file 17.
The object id in file 0x3 does not appear in the object
id index in file 0x11.
Inserting an index entry into index $O of file 17.
Creating reparse point file.
Inserting an index entry into index $I30 of file 11.
Creating index $R for file 18.
Creating quota file.
Inserting an index entry into index $I30 of file 11.
Creating index $O for file 19.
Creating index $Q for file 19.
Inserting default quota record into index $Q in file 19.
CHKDSK is verifying security descriptors (stage 3 of 3)...
24 file SDs/SIDs processed.
Cleaning up 27 unused index entries from index $SII of file 0x9.
Cleaning up 27 unused index entries from index $SDH of file 0x9.
Cleaning up 27 unused security descriptors.
Security descriptor verification completed.
Inserting data attribute into file 0.
4 data files processed.
The MFT mirror is different from the MFT.
Correcting errors in the Master File Table (MFT) mirror.
Correcting errors in the master file table's (MFT) DATA attribute.
CHKDSK discovered free space marked as allocated in the
master file table (MFT) bitmap.
CHKDSK discovered free space marked as allocated in the volume bitmap.
Windows has made corrections to the file system.
356854780 KB total disk space.
16 KB in 8 indexes.
0 KB in bad sectors.
76868 KB in use by the system.
65536 KB occupied by the log file.
356777896 KB available on disk.
4096 bytes in each allocation unit.
89213695 total allocation units on disk.
89194474 allocation units available on disk.
Internal Info:
10 00 00 00 10 00 00 00 0b 00 00 00 00 00 00 00 ................
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
------------------------------------------------------------------------------
Checking file system on E:
The type of the file system is NTFS.
Volume label is Backup.
One of your disks needs to be checked for consistency. You
may cancel the disk check, but it is strongly recommended
that you continue.
Windows will now check the disk.
Unable to query LCN from VCN 0x9 for attribute of type 0x80.
The non resident attribute of type 0x80 is inconsistent. The valid data
length is 0x6a40000, file size 0x6a40000, and allocated length 0x4000.
The non resident attribute of type 0x80 is inconsistent. The valid data
length is 0x6a40000, file size 0x4000, and allocated length 0x4000.
CHKDSK is verifying files (stage 1 of 3)...
Deleted corrupt attribute list entry
with type code 48 in file 0.
Unable to find child frs 0x24 with sequence number 0x5.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to find child frs 0x26 with sequence number 0x2.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to find child frs 0x25 with sequence number 0x1.
The attribute of type 0x80 and instance tag 0xa in file 0x0
has allocated length of 0x6a40000 instead of 0x4000.
Deleted corrupt attribute list entry
with type code 128 in file 0.
Unable to locate attribute with instance tag 0xa and segment
reference 0x1000000000000. The expected attribute type is 0x80.
Deleting corrupt attribute record (128, "")
from file record segment 0.
16 file records processed.
File verification completed.
0 large file records processed.
0 bad file records processed.
0 EA records processed.
Correcting file name errors in system file record segment 0.
0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
An index entry of index $I30 in file 0x5 points to file 0xcc1
which is beyond the MFT.
Deleting index entry $RECYCLE.BIN in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd8f1
which is beyond the MFT.
Deleting index entry .DS_Store in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xf06
which is beyond the MFT.
Deleting index entry .Trashes in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1784c
which is beyond the MFT.
Deleting index entry .windows-serial in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17328
which is beyond the MFT.
Deleting index entry 3DReaperDX in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4eb
which is beyond the MFT.
Deleting index entry bittorrent in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x103dd
which is beyond the MFT.
Deleting index entry books in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4e9
which is beyond the MFT.
Deleting index entry Downloads in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xd4e9
which is beyond the MFT.
Deleting index entry DOWNLO~1 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x11
which is beyond the MFT.
Deleting index entry found.000 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x12337
which is beyond the MFT.
Deleting index entry found.001 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x10
which is beyond the MFT.
Deleting index entry Games in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xf10
which is beyond the MFT.
Deleting index entry img in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xfe4a
which is beyond the MFT.
Deleting index entry log2.txt in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x118a0
which is beyond the MFT.
Deleting index entry mom in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1779e
which is beyond the MFT.
Deleting index entry photos in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17746
which is beyond the MFT.
Deleting index entry shufa in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17753
which is beyond the MFT.
Deleting index entry substance-designer-tut in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbd
which is beyond the MFT.
Deleting index entry System Volume Information in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbd
which is beyond the MFT.
Deleting index entry SYSTEM~1 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x23
which is beyond the MFT.
Deleting index entry TheSims3 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xcbf
which is beyond the MFT.
Deleting index entry tmp in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x17772
which is beyond the MFT.
Deleting index entry tools in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x11cd7
which is beyond the MFT.
Deleting index entry universities.txt in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xc37e
which is beyond the MFT.
Deleting index entry videos in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0xdb6c
which is beyond the MFT.
Deleting index entry videos2 in index $I30 of file 5.
An index entry of index $I30 in file 0x5 points to file 0x1731b
which is beyond the MFT.
Deleting index entry work-projects in index $I30 of file 5.
Index entry $ObjId of index $I30 in file 0xb points to unused file 0x19.
Deleting index entry $ObjId in index $I30 of file 11.
Index entry $Quota of index $I30 in file 0xb points to unused file 0x18.
Deleting index entry $Quota in index $I30 of file 11.
22 index entries processed.
Index entry $Reparse of index $I30 in file 0xb points to unused file 0x1a.
Deleting index entry $Reparse in index $I30 of file 11.
Index entry $RmMetadata of index $I30 in file 0xb points to unused file 0x1b.
Deleting index entry $RmMetadata in index $I30 of file 11.
Index entry $UsnJrnl of index $I30 in file 0xb points to unused file 0xcbc.
Deleting index entry $UsnJrnl in index $I30 of file 11.
Index verification completed.
0 unindexed files scanned.
0 unindexed files recovered.
Creating object id file.
Inserting an index entry into index $I30 of file 11.
Creating index $O for file 17.
The object id in file 0x3 does not appear in the object
id index in file 0x11.
Inserting an index entry into index $O of file 17.
Creating reparse point file.
Inserting an index entry into index $I30 of file 11.
Creating index $R for file 18.
Creating quota file.
Inserting an index entry into index $I30 of file 11.
Creating index $O for file 19.
Creating index $Q for file 19.
Inserting default quota record into index $Q in file 19.
CHKDSK is verifying security descriptors (stage 3 of 3)...
24 file SDs/SIDs processed.
Cleaning up 27 unused index entries from index $SII of file 0x9.
Cleaning up 27 unused index entries from index $SDH of file 0x9.
Cleaning up 27 unused security descriptors.
Security descriptor verification completed.
Inserting data attribute into file 0.
4 data files processed.
The MFT mirror is different from the MFT.
Correcting errors in the Master File Table (MFT) mirror.
Correcting errors in the master file table's (MFT) DATA attribute.
CHKDSK discovered free space marked as allocated in the
master file table (MFT) bitmap.
CHKDSK discovered free space marked as allocated in the volume bitmap.
Windows has made corrections to the file system.
356854780 KB total disk space.
16 KB in 8 indexes.
0 KB in bad sectors.
76868 KB in use by the system.
65536 KB occupied by the log file.
356777896 KB available on disk.
4096 bytes in each allocation unit.
89213695 total allocation units on disk.
89194474 allocation units available on disk.
Internal Info:
10 00 00 00 10 00 00 00 0b 00 00 00 00 00 00 00 ................
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
------------------------------------------------------------------------------
The data I wish to recover is around 70GB in size. One directory occupying 30GB with somewhere around 100,000 images of varying size (5KB - 16MB, usually several hundred KB in size), and two others with 40GB of video files of varying size (up to 3GB).
The filesystem is probably moderately fragmented since it's almost full and files are frequently added/removed.
I have not modified anything in the partition, but I can't guarantee Windows and the applications in background didn't do anything.
I tried recovering data with GetDataBack NTFS 4.32 yesterday, in "systematic file system damage" mode. ("Undelete" mode doesn't display the lost files, by the way.) 7 filesystems are found, 3 of them are broken, 4 of them work in a certain degree. I'm able to get half of the filesystem tree (some directories and files are missing), with correct metadata, however after extraction I found many files got apparently corrupted contents (missing JPEG SOI marker, missing FLV header, invalid video frames, etc.).
I'm playing with ZAR (Zero Assumption Recovery) 9.2 right now. Got basically the same "half" directory tree as GetDataBack. ZAR's validation detected thousands of invalid files (I believe by validating file signature). But seemingly at least the number of broken files is probably lower than the amount I got from GetDataBack.
I have not discovered clear signs of bad sectors or hardware failure on the hard drive so far, and the hard drive is about 1.5 yr in age. I don't think there's any damage to the GPT partition table, either.
The most recent backup are from like two years ago (on my old computer)... And I don't have the hashes of files, either, so I could only validate the files by viewing them manually or with the file signatures.
Any advices? Thanks in advance.
By the way, I guess I will switch to ext4 for storing those files in the future... The accident gave me a terrible on the reliability of NTFS -- why would a simple power loss kill a filesystem?
Facebook
Twitter
Instagram