PreCaching for HDDs (and SSDs)

PreCaching for HDDs (and SSDs)

Filed under: Storage - Hard Drives

I've finally gotten around to writing up a small pre-caching application for those people still using HDD's. Such as schools, or any organisation or individual that automatically boots Personal Computers and leaves them unattended for a brief period prior to use, usually by the public, and is not happy with the responsiveness of the machines - even 20 minutes or more after boot up.

This application will also help those still using mechanical Hard Disk Drives by enabling the end user to pre-cache small files on their HDD while not reading ridiculously large files that would force small files out of the OS Disk Cache.

The idea is to increase the performance when small files are accessed (not just those that make up the Start Menu) by having the Windows OS read their contents into it's own Disk Cache system.

The application traverses the file system of the C:\ drive looking for files under a certain size (64KB currently) and reads them, causing them to be cached by the Windows Operating System Disk Cache.

The application runs with the same permissions as the logged in user and only performs I/O Reads. So some files/folders will not be cached. (Usually a very small number).

Current version: 1.0.0.5
Last update date: 6th Jan 2013
Download: http://scottdbowen.id.au/PreCacher/PreCacher.exe
9 answers Last reply
More about precaching hdds ssds
  1. Benchmark
    - Using Samsung 830 SSD (128GB) on Intel Core i5 system with encryption.
    - Total Small Files PreCached: 76,838
    - Total File Content Cached: 882,922,403

    Initial pre-cache time: 124.5574 seconds
    Subsequent read-back: 13.02602 seconds



    Difference: > 9.56x the performance once cached (relative to an SSD).

    For a mechanical Hard Disk Drive (HDD) the performance gain would be even greater!
  2. Don't have a computer with HDD as OS drive to test on :(
  3. Thanks for the helpful contribution; I'm testing it now. Should it be in one's startup folder? Does it run continuously in the background?

    John LeBlanc

    Scott_D_Bowen said:
    PreCaching for HDDs (and SSDs)

    Filed under: Storage - Hard Drives

    I've finally gotten around to writing up a small pre-caching application for those people still using HDD's. Such as schools, or any organisation or individual that automatically boots Personal Computers and leaves them unattended for a brief period prior to use, usually by the public, and is not happy with the responsiveness of the machines - even 20 minutes or more after boot up.

    This application will also help those still using mechanical Hard Disk Drives by enabling the end user to pre-cache small files on their HDD while not reading ridiculously large files that would force small files out of the OS Disk Cache.

    The idea is to increase the performance when small files are accessed (not just those that make up the Start Menu) by having the Windows OS read their contents into it's own Disk Cache system.

    The application traverses the file system of the C:\ drive looking for files under a certain size (64KB currently) and reads them, causing them to be cached by the Windows Operating System Disk Cache.

    The application runs with the same permissions as the logged in user and only performs I/O Reads. So some files/folders will not be cached. (Usually a very small number).

    Current version: 1.0.0.5
    Last update date: 6th Jan 2013
    Download: http://scottdbowen.id.au/PreCacher/PreCacher.exe
  4. You can place it in your StartUp folder if you want and are willing to live with a 60 second to 5 minute 'grind' of slightly heavier HDD I/O during login; or ideally leave the PC idle and go make a coffee.

    Facts about v1.0.0.5 of PreCacher.exe

    - The current version caches all files that are 64KB or less as well as most of the MFT which reduces later mechanical I/O significantly.
    - This works out to pre-caching about 70% of all the files on a typical C: drive within a space of 680MB to 1024MB. (Yes, that's right: 70% of your files on C: -by quantity- will probably fit within 1GB, so long as it's just the smaller ones).

    - I am testing a version that pre-caches all files under 96KB, 128KB, 160KB, 192KB, 224KB, 256KB, etc.

    - It does not run in the background. (I may release a Service version that just refires the service every 30 minutes or so down the track).

    - It may benefit from being scheduled as a Task run every 15 minutes at most (quad core) to every 60 minutes at most (single core).

    - A subsequent run after it has cached should finish in about 1/30th the time (give or take).

    - It does not add, delete, create or modify any registry entries (Although the .NET Installation method might add some registry entries just for it).

    - It does not change the way that the Windows® 2K/XP/Vista/7/etc. Operating System Disk cache functions in any way, it merely leverages code that Microsoft® have already bundled with the Windows® OS.

    - It works on systems with ReadyBoot/ReadyBoost/PreFetching enabled or disabled in any combination.

    - Some anti-virus products with real-time scanning might make the window appear as 'Not Responding', I am currently investigating one potential case of this.

    - Both Cache (Read, not Write Back) and Free memory are in the 'Available Memory' pool in Windows Vista® and Windows 7®, each are immediately available for allocation by running software.

    - Loading an application that suddenly allocates all 'Available' memory (both Cached and Free, if any) will negate the performance gain as read-cache data is forced out of the cache as the Windows OS disk cache shrinks to under 128MB in size.

    - Re-running after such an event will resolve the above however.

    - The data in C:\Windows\Prefetch\. and C:\Windows\Prefetch\ReadyBoost\. (if enabled) will slowly be retrained and the ReadyBoot/ReadyBoost/PreFetch trace ETL 'log' files will likely grow to 20MB which is completely normal. (I mention this as some overzealous System Admins may want to know this in advance; and it is better to be slightly overzealous about such things IMHO).

    That's pretty much everything I can think of for now.

    I've updated some doco and added it to the folder on my web-server.
    - http://scottdbowen.id.au/PreCacher/

    Enjoy.


    PS: If you only have 1GB of RAM let me know as I'm willing to release versions that tune themselves to specific machines. The current version caches all files that are 64KB or less as well as most of the MFT which reduces later mechanical I/O significantly.

    I believe that machines with 1GB (or less) of system memory have ReadyBoost disabled --- unless they used to have 2GB and 1GB was removed!
  5. PreCacher v1.0.1.6 released
    - Some minor big fixes.
    - If the PreCache button is clicked again after the initial pre-cache the timer gets stopped then restarted afterwards.
    - The application now closes after 30 seconds of inactivity after displaying its report. Instead of 60 seconds.
    - v1.0.1.6 attempts to cache all files under 96KB in size on C: drive using a 4KB buffer (might be faster on some machines, might be slower, I doubt anyone will notice the difference).
    - Report enhanced slightly to include the percentage of files read.

    *** Time taken: 13.65203 seconds.
    *** Files: 104711	(Denied: 248)
    *** Directories: 23913	(Denied: 248)
    *** Total Small Files PreCached: 85136 of 104711
    ···      ~81.30569% of all files on C: drive!
    *** Total Read: 1287 MB
    *** Speed: 101402KB/sec
    *** Automatically closing in 5 seconds.
    *** Automatically closing in 4 seconds.
    *** Automatically closing in 3 seconds.
    *** Automatically closing in 2 seconds.
    *** Automatically closing in 1 seconds.
  6. PreCacher v1.0.2.0 LE released (for machines with only 1GB of RAM)
    - Some minor big fixes.
    - If the PreCache button is clicked again after the initial pre-cache the timer gets stopped then restarted after being reset to 30 seconds again. [:-P].
    - v1.0.2.0 LE only attempts to cache all files under 12KB in size on C: drive using a 4KB buffer (might be faster on some machines, might be slower, I doubt anyone will notice the difference).
    - Report enhanced slightly to include additional information & the percentage of files read.

    *** PreCacher v1.0.2.0.LE for 1GB RAM PCs, reading all files under 12 KB
    *** Time taken: 13.08602 seconds.
    *** Files: 104738	(Denied: 248)
    *** Directories: 23929	(Denied: 248)
    *** Total Small Files PreCached: 57160 of 104738
    ···    ~54.57427% of all files on C: drive!
    *** Total Read: 200 MB
    *** Speed: 15808KB/sec
  7. To anyone that see's this, I am going to write up a non-blocking version (v2.x.x.x and greater) to resolve some issues --- hopefully this weekend and/or next weekend!

    I've got enough of cross-threading down pat:
    - http://scottdbowen.id.au/CrossThreadDemo/

    It should resolve the remaining issues that most people were having...

    I've removed most of the versions at: http://scottdbowen.id.au/PreCacher/ pending this new version

    EDIT: PreCacher v2.0.0.0 has now been released!
  8. Scott_D_Bowen said:
    To anyone that see's this, I am going to write up a non-blocking version (v2.x.x.x and greater) to resolve some issues --- hopefully this weekend and/or next weekend!

    I've got enough of cross-threading down pat:
    - http://scottdbowen.id.au/CrossThreadDemo/

    It should resolve the remaining issues that most people were having...

    I've removed most of the versions at: http://scottdbowen.id.au/PreCacher/ pending this new version

    EDIT: PreCacher v2.0.0.0 has now been released!


    Links are dead
  9. BuBu Shofronea said:
    Scott_D_Bowen said:
    To anyone that see's this, I am going to write up a non-blocking version (v2.x.x.x and greater) to resolve some issues --- hopefully this weekend and/or next weekend!

    I've got enough of cross-threading down pat:
    - http://scottdbowen.id.au/CrossThreadDemo/

    It should resolve the remaining issues that most people were having...

    I've removed most of the versions at: http://scottdbowen.id.au/PreCacher/ pending this new version

    EDIT: PreCacher v2.0.0.0 has now been released!


    Links are dead

    Here if anyone wants this, I downloaded v2 a long time ago and just came across it while going through my backups.

    https://www.dropbox.com/s/v87jfhcyys713xb/PreCacher_v2000.exe?dl=0 - standalone
    https://www.dropbox.com/s/rbg4k5e8cku0ulv/PreCacher_v2000_Installer.zip?dl=0 - installer
    https://www.dropbox.com/s/ux4gf64qfm23vrl/ReadMe.rtf?dl=0 - readme
Ask a new question

Read More

Hard Drives SSD Cache Storage