Storage/Backup - A Primer

Hi All

I've spent some time researching and executing my storage & backup configuration and am sharing it so others can benefit and/or make further recommendations.

BACKGROUND:

While not my profession, I am a photographer with close to 2TB of images, both digital capture and negative scans. In addition I have 500g of music files, about 2TB of backed up DVDs and 500g of general data files, including OS, email etc. That's about 5TB.

My image files are my single most treasured possession and represent my life memories/art and are irreplaceable.

Over the years I have used various approaches to storing/backing-up/verifying the above. Along the way, I have had my share of crashed drives and lost material, fortunately excluding any image files.

For the 1st time, I am now comfortable with my storage/backup configuration. For this reason I am writing this post in case it can help others, or others can help me improve it further.

NOTE: I am NOT an IT professional, just a user with lots of important files I need constant access to and want to preserve for eternity. My methodology may not be industry standard and my descriptions/terminology/assumptions/conclusions may be imprecise. However, hopefully you will find useful info here that will save you lots of time, and moreover ensure your data is always safe. It is in that light that I write this...

REQUIREMENTS:

-- Ample storage
-- Extremely solid backup(BU)
-- Multiple BUs
-- On and off site BU
-- Data verification and integrity checks
-- More than one BU system (in case software is not as good as advertised)
-- Relative simplicity
-- Automation to ensure ongoing maintenance
-- Minimal system performance sacrifice
-- Reasonable cost

IMPLEMENTATION:

(Note: this is what works for me. I don't believe anyone needs to duplicate this config, as each part may be useful alone and others may prefer modification/alternative solutions for the individual parts e.g Mac vs PC, alternative hardware/software/service-provider)

Based on the above, this is my configuration :

OS/PROGRAMS:

Windows 7 64 bit (this has nothing to do with my config, its just what I use). OS/Programs are on a SSD (Solid State Drive) with no other data on it. "User Accounts", "Application Data", "App Data", "Temp Folder", "Paging File", "Recovery Snapshots", Outlook "PST/OST" file, Itunes Library, etc, etc are relocated to a separate drive. Without getting into too much detail on this, the point is to keep as little as possible on the OS SSD drive for both speed and capacity. Even if others don't relocate most of what is describes above to another drive, simply having a SSD for your OS/Program ("C" ) drive makes unbelievable improvement on performance, vs traditional drives.

PRIMARY STORAGE:

4 x 2TB high performance consumer HDDs (Western Digital Black 7200 RPM, 64mb cache 2TB). I prefer fast internal drives over NAS for my primary storage. I do this primarily for performance reasons, but also for simplicity/convenience. Fast internal drives will out perform a NAS. These drives are about $200 each. Certainly for all the directories described above (which are accessed constantly) and for my photography files, which required fast drive performance for Lightroom and Photoshop, the WD Black drives are great. For general data files, Media, etc, any reasonable HD should be fine and some may prefer the poser savings of "Green" drives (these power down when not in use and spin at 5400 rpm to significantly save power. 2TB WD-EARX is about $130 if you shop).

Just for convenience (and ease of back up described below) I organize my 4 storage drives as follows:

--Drive 1: General data files and directories I remap from the OS/Program drive ("User Accounts", "Application Data", "App Data", "Temp Folder", "Paging File", "Recovery Snapshots", Outlook "PST/OST" file, Itunes Library, etc, etc)
--Drive 2: Photography files, Lightroom catalog, etc
--Drive 3: Media
--Drive 4: Archive (more on this below)

PRIMARY BACKUP:

For my 1st backup location I use an older 4-bay NAS (Netgear ReadyNas 600) in RAID 5 configuration. With 4 drives, Raid 5 yields 3 drives worth of storage size and allows one drive to fail without losing any data. Because speed is not important here (because I am only backing up to the NAS), I use the WD Green drives described above for price and energy use reasons.

NOTE: There is a question of whether "green" drives are ideal for NAS use. The reason is, that these drives spin down to conserve energy and some NAS's (or maybe all NAS's, sometimes) think the "sleeping" drive has faulted, causing the NAS to rebuild the data on what it thinks is a faulted drive. Based on my research, there seems not to be a consensus about this. If cost and energy use is not important to you, possibly consider non-green drives.

Most people seem to use their NAS for primary storage and then back up the NAS. For the reasons above, I do this in reverse.

Note: RAID by itself is NOT safe!!! Meaning, Raid can fail, so if you have Raid you still need to back up, just like with any other drive configuration. Raid is less likely to fail than a single drive, and in Raid more than one drive can fail before data loss, but they can still fail/be stolen/be damaged, etc, etc. If you have Raid,you still need to back up!

SECONDARY/OFF-SITE BACK UP:

If you have critical files (for me, my photography files) or just important ones (for me, my music. OS/Program drive and general data files) off-site and secondary back up is both important and easy. Its also cheap!

There are two basic ways of doing this: The traditional way is to back up to a 2nd set of drives and physically store them off site. If you do this, the most you will lose if your house burns down :-( is the data that changed since your last back up to the off-site drives. The problem is you need to actually do this, most wont after a while.

The far better way is to use a cloud storage provider. I use CrashPlan. They offer UNLIMITED STORAGE CAPACITY FOR $3/MONTH!! (for a single PC based on a 4 year plan (a whopping $139) or $5/mo on a month-to-month plan. An unlimited capacity family plan covers up to 10 computers and costs $6/mo on a 4 year plan ($287 total) or $12/mo month-to-month. They also offer a free plan with which you can back up to another PC, including one off-site. Using that option, just buy your friend/mother/boss/kid whatever size drives you need and back up for free to your drives at their location.

There are many other very affordable cloud storage solutions. For me, CrashPlan was best given my capacity needs and other features they offer. Some of the features I likes are:

-- Can send them a drive up to 1-2 TB?? with your data on it, so don't need to "seed" all of this online. Then only new/changed files are updated.
--Can have them send a HDD 1-2TB?? back to you if you crash and need to restore. That's a lot faster than downloading the restore data. Cost is about $100 for either disk option, if and when used.
-- Can select any directories/sub directories and/or portions thereof, from any of your drives to back-up. Some services need a single root folder. This flexibly was critical for me, since I have many drives.
--Have Iphone/Ipad App, to access all of your online data and download to phone/pad (very handy)
--Supports "versioning" meaning you can restore any version of any file. This means that if you change or delete a file, you can restore the backed up copy from any date you choose. This is invaluable if you need something you deleted or changed.
-- Real-time BU, meaning files are monitored and backed up continuously (can set to schedule instead of real-time, if preferred)
--Well reviewed with national clients, meaning its safe, secure and trusted.

NOTE: I have no affiliation with Crash Plan or anyone involved with it. I am simply very happy with it.

NOTES ON SOFTWARE BACKUP METHODS:

Note on DATA VERIFICATION/INTEGRITY CHECKS: Most programs (including Windows/Apple copy & paste, as well as many backup/sync programs do not either 1) truly verify that files are/aren't the same, nor 2) ensure that the copy made is identical before overwriting the destination file:

1) Most programs only compare files by Name/Size/Date-stamp. That means that is a file is corrupted the program has no way of knowing and assumes is it unchanged. A relative few programs use Name/Size/Date-stamp AND CRC hash checks (DM5 Sums)that compare every bit of file data between two files to determine that they are truly identical (bit perfect). This same process can also be used to determine if files have changed (become corrupted) over time. So for example if you have an archive drive of your music/photos, etc you stored a year ago, that drive and the files can become corrupt. This same data verification process can place a DM5 check sum file on the drive so you can later verify it (by having the program recalculate the check sum of the drive contents and comparing it to the stored check sum file). It would then be able to tell you exactly which files have been corrupted.

This type of compare/verify is slower than Name/Size/Date-stamp and is not always appropriate for the task, but when it is, you want to have it!

2) When copying a file ( i.e. backup/sync, etc) better programs make "Safe Copies" and then change the name after determining the copy was perfect. This means that when backing-up/syncing between two drives, where an existing file was updated on one side, the program will copy the file to the other location, but not overwrite the existing (older) file on the destination until the new copy is determined to be perfect. Then it renames it and removes the older file. This means that if something bad happens during a copy/sync/backup process no existing files get lost or damaged.

Note on FILE COPY VS FILE IMAGE: There are two basic types of backups, 1) Copies and 2) Images:

1) Copies: This is your normal file/folder copy.

2) Image Copy: The copy of any number of files/folders is contained within a single backup "image". You cannot open this directly (but can mount the image and it will look like another drive/cd and can be browsed). You can also restore some or all of the contents of the image back to normal files/folders.

When using Image type BU programs like these, one has 3 basic choices: 1) Full BU = a complete new image of ALL the data being backed-up; 2) Differential = after the 1st full back up, all subsequent BUs are of new/changed files since the last full backup. In this, the full BU plus the selected differential BU equal you current state. 3) Incremental BU = after the initial full back up, all subsequent BUs are of new/changed files since the last incremental BU. In this case the original full BU plus ALL incremental BUs equal the current state. I have my BU schedule set to do a variety of these, so I have periodic full BUs, plus incremental BUs every few days, replaced by differential BUs monthly. For me this gives me the desired history but does not require numerous images be preserved for a full restore.

Most normal files can be backed up in either way. Your operating system and programs can only be backed up as image files. Most back up programs back up one way or the other, but not both ways.

I have no ideal if one or the other is considered "better" (except for your OS/Programs, where an image is the only way).

I prefer the normal file/folder back up, because I can see, touch and smell (figuratively)the files, just as they existed on the source. Also, if I need to restore, I can just swap the back up with the prior source and I'm done (but would also want to back up again in case this drive fails).

BACK-UP/SYNC ROUTINE:

Now we get to what to(I)do...

There are many ways to do this. As listed above under Requirements, however you do it, I recommend:
-- Data Verification and Integrity checks
-- Automation
-- More than one software (in case doesn't work as expected)

1) Backup from Primary Storage to NAS (or other local BU location): I do "normal" vs "Image" backups of all my primary storage to my NAS. I use SyncBack Pro http://www.2brightsparks.com/syncb [...] tures.html and/or ViceVersa Pro http://www.tgrmn.com/ to do these as verified copies and MD5 hash check comparisons and sync my NAS with all of my primary storage. If the verified copy and hash check process takes too long, I may set the program to do non verified copy methods and periodically run data verification (weekly, monthly/yearly). (Note: I've recently set up the NAS so I dont know if the safer verified BU will run fast enough. Previously, I just used 4x2TB external HDDs and backed up each of my 4 internal HDDs to individual drives. As these were all SATA connections the verified BU speed was not an issue)

I set 4 backup plans (very easy to do in any BU program), one for each primary drive (to give me better control eg settings, schedule, verify or not, etc on a per drive basis)and use the schedule feature to run each every few days at 3am. Note: The only thing that will be copied/deleted on the NAS are those source files that have changed since the last back up, so its not lots of writing per back up. NOTE: I do not back up my OS/Program drive this way, see below:

2) Backup of OS/Program Drive: Because this needs to be done as an "Image" BU, and the above programs don't do that, I use either Acronis True Image Home http://www.acronis.com/homecomputi [...] trueimage/ or EASEus ToDo Backup (free) http://www.todo-backup.com/product [...] ftware.htm . I used to use Acronis, but find their interface very frustrating and now use EASEus. Both do the same thing with small differences that seem to be just personal preference. Both programs can be set to Verify the image they created and also to re-verify older images on whatever schedule you desire. They can copy entire drives/partitions or whatever files/folders you select. In either case their BU is an Image file.

I image my OS/Program drive and save that to my 4th primary drive (archive). This then gets backed up to the NAZ as described above (possibly over kill, but I have the space).

NOTE: Because as stated above, I prefer more than one software to avoid surprises on a restore (likely just because I don't know enough to trust one solution, I also make an image BU of my normal data files, including My docs, etc, etc (again because I have the space). This is first saved to my Archive drive and backed up to NAS. This Image BU is in addition to the "Normal" file/folder BU , ie these files get backed up both as Images and Normal copies.

3) Off-site Backup: As described above, I use CrashPlan unlimited storage plan. I could either back up the NAS (because it holds the local backup of all my primary drives) or back up the primary drives directly. I do the latter, feeling its best to back up from the source than the backup. This also limits traffic to the NAS, which is slower than my main system. CrashPlan runs continuously in the backround. It does not slow my systen al all. It can slow my internet if Im doing heavy downloads, video streamming, etc. If that bothers me, I suspend crachplan as needed. You can also set times for it to run, if you prefer.

One could set a single backup routing for CrashPlan, but I prefer to have one for each primary drive. This way, if I want to prioritize, I can have one run and pause the others. CrashPlan is doing realtime back up (but may lag due to upload/download limits is a lot of files have changed).

I hope this sparks some thoughts and part of this is helpful to others. I will edit this, but for now need to go so no proof read.

Thanks

-- Caleb