Sign in with
Sign up | Sign in
Your question

Strange data corruption with large files

Last response: in Apps General Discussion
January 24, 2009 11:01:34 PM

I have 4 machines.
1 windows 2000 server with sata-ii drives
1 xp laptop sp2
1 xp laptop sp3
1 vista laptop sp1
All avg pro v8

When I copy a 10gb file from any machine to any other machine(copy/xcopy) and follow this with a "fc /b" all copies are identical.

However.... if I use ftp, rsync, or any other delta copying method I always get errors from "fc /b", usually 2 bits (02 -> 00) sometimes a few more.

When I overwrite the same file with copy /b or xcopy or even xxcopy, no problems, 100% ok.

Since this is happening to 4 different OS's and hardware systems I am excluding bad HD's and ram.

I have tried this with a smaller file like 2gb, no problems here.

I've been running tests for days now, recompiling rsync and other delta tools, added extra layers that run a md5 and sha1 checksum (who always report that the data written to disk matches both checksums) and still fc detects at least 2 wrong bits. xxcopy again and all is well.. slightly going insane here...

Any ideas?
January 26, 2009 7:05:49 PM

Nobody any ideas? could someone do a test?
Ie. take a DVD and copy this as a ISO to PC-1
take another DVD and copy this as a ISO to PC-2
Install something like deltacopy and (r)sync one ISO with the other between pc's.
Then run a fc /b between pc's ISO's.
January 28, 2009 7:14:51 AM

if the file still works then the extra two bits are probably just header information that the ftp program has put there.... don't worry about it.
Related resources
January 28, 2009 8:23:01 AM

I wish... the bits are located at offsets like ecd09981 ie. anywhere beyond the 2.5g range. and no its not a 16/24bit limit problem.
January 30, 2009 11:12:23 AM

Just a quick twitter, I now suspect windows folder compression, its the only thing commonly used between pc's.
February 2, 2009 11:02:49 AM

Yep, confirmed, its windows disk compression doing this on all platforms, so not just a bug in home server, compress files larger then 1gb and you risk corruption... it took a few days to test this with 400gb, decompressing and repeat.