How to transfer many files ~10^7 quickly between drives

You could Ghost clone the drive. That only an option if there is nothing else on the destination drive. I've found that copying large numbers of files is best done in blocks of folders due to the file structure being extensive.

Your problem is the number of files you are trying to transfer. You would be right to expect faster rates if you were transferring say a few HD movies. It is all in how things are read from a mechanical HDD. Large HD movies are sequential, in that one memory block is generally right after the one before it.

You however are transferring much smaller files. Individual files have no guarantee to be sequential, most of the time they aren't sequential. Your guess about fragmentation is partially correct but even defragmenting your HDD isn't going to help. You still have no guarantee that the files will be read sequentially.

After every file is read your HDD has to wait to reposition the head to find the next file. So what you are running into is the worst case scenario for HDDs.

Cloning the disk would be a faster way of doing it because there will be much less head movement and much more sequential reading.

Best answer selected by Jsplinter.