best way to backup Very large media every day.

OffbeatBryce

Honorable
Mar 27, 2017
138
3
10,685
Hello,

I run a film business and I film hours and hours of footage a week. I have roughly over 1TB sometimes of one film project per month as I make documentaries. I also sometimes shoot in 4K.

I've tried Backblaze and other online backup solutions but here are my problems.

with Backblaze running for a month with the computer not going to sleep still has an estimated 100,000 files left to backup. Not a very good solution for my film business as I just keep adding more and more files. My fear is that if a hard drive fails backblaze wouldn't have backed up everything in time. I checked and so far it's only backed up 1TB out of 5 and it's been one month with the computer on 24/7 and without it going to sleep. I have sleep turned off. I also have 3 other drives with another 5 TB.

I can't afford 90 a month for larger storage for Google Drive and One Drive only limits it's storage to 1TB.

What do I do?

I've heard about M Discs but is that really a solution? I mean some people claim M Discs last 1,000 years but what if in like 20 years DVD drives don't exist? Also can any burner write to M Discs?

I've thought about off site storage but the problem with that is you either risk a fire or flood etc.

What exactly do I do here?

EDIT: I generally have anywhere from 500GB-800GB a day in video between me filming footage and also obtaining footage from other clients to edit. I also want to mention that Backblaze backs up on average 5GB per hour. My internet connection slow and I can't afford a faster connection due to where I live.
 
Solution
Most of the mainstream online cloud storage are not designed for high high file size usage. They are designed to keep your excel documents synced up at a reasonibel speed.

At 50mbps you have a speed limitition so you have too look at how much Bytes of data you need changed evey day. If you have 1TB of changed data daily that would take 44 hours a day to backup assuming that 50mbps was also uplaod AND that you could sustain it. This is clearly not going to work.
Now if you say have 100GB a day of changed files then having a local backup and an offiste backup would be doable at 50mbps UPLOAD speed.

If your upload speed is much lower then that or your changed file size is greater then that then you have no other option but to do your...

Eximo

Titan
Ambassador
You do need specific M-disc compatible drives. If the entire idea behind the storage medium is longevity you can bet someone is going to be lined up to be making optical disc readers for a long while. Even if not, I can still go out and buy a floppy drive, so it is not like twenty years is too long for a legacy tech to stick around.

I'm curious what your connection rate it is that you haven't chipped away at 5TB very quickly. If this is the first backup then that isn't the worst thing. Afterwords it would become differential.

Unless offsite storage is a must, then a local NAS might be your best option. You'll want decent drive performance from both your client and storage and a good network connection.

Using SSDs as your working cache folder might be an ideal solution. 1TB SATA SSDs are quite reasonable now.
 

OffbeatBryce

Honorable
Mar 27, 2017
138
3
10,685
I have 50mbps Internet. I've read that others who have used backblaze have the same issue with a faster connection than mine. Some guy has let it backup 100GB a day for a year and it's still not halfway done. Even if the inital backup takes a while it shouldn't take a year.
 

OffbeatBryce

Honorable
Mar 27, 2017
138
3
10,685





but lets say there is a fire off site. Then what do I do? I've heard of Fire Proof boxes but don't those only tolerate heat for papers and not drives?
 

Lee-m

Distinguished
Jan 27, 2009
866
0
19,210

Then you still have your originals on site. Thats what a backup is, additional copies.

If you go the local nas route, you should still store off site backups. My recommendation was just a cheaper version of a local nas.

That would be my recommendation anyway. There are better ways to handle it, but they all come at additional costs, and you have said that would be an issue, and uploading such big amounts of data isnt really an option.

 

RolandJS

Reputable
Mar 10, 2017
1,230
21
5,715
A very large capacity NAS [not RAID - according to some sources], a box 'o high-quality, platter-driven, mega-GB-monstor-sized, hard-drives might be a great benefit to you. I guess you can come home or to your office after field work and kick in the NAS and kick in the backup process. Because these files are your money-making files, do not go cheap or EZ-process; upfront quantity monies for upfront quality backup process will serve you well. Concerning fires, floods, etc., hey - you drive right? You bring some of your stuff along with you, right? Well, hopefully not, however, bad natural and man-made things can happen to you and your stuff on the road, in the field -- however, your making a living and your backing up your living trumps all else.

[I see eximo and I posted at the same time :) ]
 

Eximo

Titan
Ambassador
Not really sure how you are responsible for a fire offsite. I imagine there are electronics fire rated boxes, but that would involve doing a rotating backup or something and constantly swapping drives if you were to keep the drive onsite.

3, 2, 1 rule is usually wise. 3 Backups, 2 types, 1 offsite.

If your goal is data redundancy then getting a fast backup is of more a priority then offsite. So a local NAS is still a good idea. Let it take on the burden of backing up to an outside source rather than your working system if you still want to pursue that.

Depending on the data, just swapping a drive in the NAS and taking the removed drive somewhere else might be sufficient to insure some disaster protection.
 
Most of the mainstream online cloud storage are not designed for high high file size usage. They are designed to keep your excel documents synced up at a reasonibel speed.

At 50mbps you have a speed limitition so you have too look at how much Bytes of data you need changed evey day. If you have 1TB of changed data daily that would take 44 hours a day to backup assuming that 50mbps was also uplaod AND that you could sustain it. This is clearly not going to work.
Now if you say have 100GB a day of changed files then having a local backup and an offiste backup would be doable at 50mbps UPLOAD speed.

If your upload speed is much lower then that or your changed file size is greater then that then you have no other option but to do your own backups to the "offisite" drive and take it home with you (I would hihgly suggest a good protective case for this).

Either way you can use syncback SE to schedule and find the different files and back them up, keep old versions of the data and if doing the network can even schedule it to initiate a transfer to an FTP server or connect via VPN. As stated by others you should have your data, your onsite backup, and your offiste backup.
Your onsite backup should also include full disk images of your computer systems which will provide an easy 30min or less recovery from windows corruptions, viruses, and drive failures.
 
Solution