Sign in with
Sign up | Sign in
Your question

PreCaching for HDDs (and SSDs)

Tags:
  • Hard Drives
  • SSD
  • Cache
  • Storage
Last response: in Storage
Share
a b G Storage
January 6, 2013 9:24:56 AM

PreCaching for HDDs (and SSDs)

Filed under: Storage - Hard Drives

I've finally gotten around to writing up a small pre-caching application for those people still using HDD's. Such as schools, or any organisation or individual that automatically boots Personal Computers and leaves them unattended for a brief period prior to use, usually by the public, and is not happy with the responsiveness of the machines - even 20 minutes or more after boot up.

This application will also help those still using mechanical Hard Disk Drives by enabling the end user to pre-cache small files on their HDD while not reading ridiculously large files that would force small files out of the OS Disk Cache.

The idea is to increase the performance when small files are accessed (not just those that make up the Start Menu) by having the Windows OS read their contents into it's own Disk Cache system.

The application traverses the file system of the C:\ drive looking for files under a certain size (64KB currently) and reads them, causing them to be cached by the Windows Operating System Disk Cache.

The application runs with the same permissions as the logged in user and only performs I/O Reads. So some files/folders will not be cached. (Usually a very small number).

Current version: 1.0.0.5
Last update date: 6th Jan 2013
Download: http://scottdbowen.id.au/PreCacher/PreCacher.exe

More about : precaching hdds ssds

a b G Storage
January 6, 2013 10:50:31 AM

Benchmark
- Using Samsung 830 SSD (128GB) on Intel Core i5 system with encryption.
- Total Small Files PreCached: 76,838
- Total File Content Cached: 882,922,403

Initial pre-cache time: 124.5574 seconds
Subsequent read-back: 13.02602 seconds



Difference: > 9.56x the performance once cached (relative to an SSD).

For a mechanical Hard Disk Drive (HDD) the performance gain would be even greater!
a b G Storage
January 6, 2013 11:29:00 AM

Don't have a computer with HDD as OS drive to test on :( 
Related resources
January 6, 2013 1:52:23 PM

Thanks for the helpful contribution; I'm testing it now. Should it be in one's startup folder? Does it run continuously in the background?

John LeBlanc

Scott_D_Bowen said:
PreCaching for HDDs (and SSDs)

Filed under: Storage - Hard Drives

I've finally gotten around to writing up a small pre-caching application for those people still using HDD's. Such as schools, or any organisation or individual that automatically boots Personal Computers and leaves them unattended for a brief period prior to use, usually by the public, and is not happy with the responsiveness of the machines - even 20 minutes or more after boot up.

This application will also help those still using mechanical Hard Disk Drives by enabling the end user to pre-cache small files on their HDD while not reading ridiculously large files that would force small files out of the OS Disk Cache.

The idea is to increase the performance when small files are accessed (not just those that make up the Start Menu) by having the Windows OS read their contents into it's own Disk Cache system.

The application traverses the file system of the C:\ drive looking for files under a certain size (64KB currently) and reads them, causing them to be cached by the Windows Operating System Disk Cache.

The application runs with the same permissions as the logged in user and only performs I/O Reads. So some files/folders will not be cached. (Usually a very small number).

Current version: 1.0.0.5
Last update date: 6th Jan 2013
Download: http://scottdbowen.id.au/PreCacher/PreCacher.exe

a b G Storage
January 7, 2013 8:01:38 AM

You can place it in your StartUp folder if you want and are willing to live with a 60 second to 5 minute 'grind' of slightly heavier HDD I/O during login; or ideally leave the PC idle and go make a coffee.

Facts about v1.0.0.5 of PreCacher.exe

- The current version caches all files that are 64KB or less as well as most of the MFT which reduces later mechanical I/O significantly.
- This works out to pre-caching about 70% of all the files on a typical C: drive within a space of 680MB to 1024MB. (Yes, that's right: 70% of your files on C: -by quantity- will probably fit within 1GB, so long as it's just the smaller ones).

- I am testing a version that pre-caches all files under 96KB, 128KB, 160KB, 192KB, 224KB, 256KB, etc.

- It does not run in the background. (I may release a Service version that just refires the service every 30 minutes or so down the track).

- It may benefit from being scheduled as a Task run every 15 minutes at most (quad core) to every 60 minutes at most (single core).

- A subsequent run after it has cached should finish in about 1/30th the time (give or take).

- It does not add, delete, create or modify any registry entries (Although the .NET Installation method might add some registry entries just for it).

- It does not change the way that the Windows® 2K/XP/Vista/7/etc. Operating System Disk cache functions in any way, it merely leverages code that Microsoft® have already bundled with the Windows® OS.

- It works on systems with ReadyBoot/ReadyBoost/PreFetching enabled or disabled in any combination.

- Some anti-virus products with real-time scanning might make the window appear as 'Not Responding', I am currently investigating one potential case of this.

- Both Cache (Read, not Write Back) and Free memory are in the 'Available Memory' pool in Windows Vista® and Windows 7®, each are immediately available for allocation by running software.

- Loading an application that suddenly allocates all 'Available' memory (both Cached and Free, if any) will negate the performance gain as read-cache data is forced out of the cache as the Windows OS disk cache shrinks to under 128MB in size.

- Re-running after such an event will resolve the above however.

- The data in C:\Windows\Prefetch\. and C:\Windows\Prefetch\ReadyBoost\. (if enabled) will slowly be retrained and the ReadyBoot/ReadyBoost/PreFetch trace ETL 'log' files will likely grow to 20MB which is completely normal. (I mention this as some overzealous System Admins may want to know this in advance; and it is better to be slightly overzealous about such things IMHO).

That's pretty much everything I can think of for now.

I've updated some doco and added it to the folder on my web-server.
- http://scottdbowen.id.au/PreCacher/

Enjoy.


PS: If you only have 1GB of RAM let me know as I'm willing to release versions that tune themselves to specific machines. The current version caches all files that are 64KB or less as well as most of the MFT which reduces later mechanical I/O significantly.

I believe that machines with 1GB (or less) of system memory have ReadyBoost disabled --- unless they used to have 2GB and 1GB was removed!
a b G Storage
January 7, 2013 11:28:28 AM

PreCacher v1.0.1.6 released
- Some minor big fixes.
- If the PreCache button is clicked again after the initial pre-cache the timer gets stopped then restarted afterwards.
- The application now closes after 30 seconds of inactivity after displaying its report. Instead of 60 seconds.
- v1.0.1.6 attempts to cache all files under 96KB in size on C: drive using a 4KB buffer (might be faster on some machines, might be slower, I doubt anyone will notice the difference).
- Report enhanced slightly to include the percentage of files read.

*** Time taken: 13.65203 seconds.
*** Files: 104711 (Denied: 248)
*** Directories: 23913 (Denied: 248)
*** Total Small Files PreCached: 85136 of 104711
··· ~81.30569% of all files on C: drive!
*** Total Read: 1287 MB
*** Speed: 101402KB/sec
*** Automatically closing in 5 seconds.
*** Automatically closing in 4 seconds.
*** Automatically closing in 3 seconds.
*** Automatically closing in 2 seconds.
*** Automatically closing in 1 seconds.
a b G Storage
January 7, 2013 12:19:08 PM

PreCacher v1.0.2.0 LE released (for machines with only 1GB of RAM)
- Some minor big fixes.
- If the PreCache button is clicked again after the initial pre-cache the timer gets stopped then restarted after being reset to 30 seconds again. [:-P].
- v1.0.2.0 LE only attempts to cache all files under 12KB in size on C: drive using a 4KB buffer (might be faster on some machines, might be slower, I doubt anyone will notice the difference).
- Report enhanced slightly to include additional information & the percentage of files read.

*** PreCacher v1.0.2.0.LE for 1GB RAM PCs, reading all files under 12 KB
*** Time taken: 13.08602 seconds.
*** Files: 104738 (Denied: 248)
*** Directories: 23929 (Denied: 248)
*** Total Small Files PreCached: 57160 of 104738
··· ~54.57427% of all files on C: drive!
*** Total Read: 200 MB
*** Speed: 15808KB/sec
a b G Storage
January 10, 2013 1:41:33 PM

To anyone that see's this, I am going to write up a non-blocking version (v2.x.x.x and greater) to resolve some issues --- hopefully this weekend and/or next weekend!

I've got enough of cross-threading down pat:
- http://scottdbowen.id.au/CrossThreadDemo/

It should resolve the remaining issues that most people were having...

I've removed most of the versions at: http://scottdbowen.id.au/PreCacher/ pending this new version

EDIT: PreCacher v2.0.0.0 has now been released!
!