Sign in with
Sign up | Sign in
Your question

Best Solution for Storing 30TB of data?

Tags:
  • Hard Drives
  • Storage
  • Business Computing
  • External Hard Drive
  • Storage Solutions
Last response: in Business Computing
Share
August 13, 2014 7:56:41 AM

I've recently been tasked with devising the best way to store data from a new x-ray system being inputted into our company. The system will be collecting and storing the data from the x-rays for a 2 year period resulting in an estimate of about 26TB of data. Cost isn't too much of an issue, but the files need to be retrievable at any time for reference. It's not really suitable for us to store this data on our current servers because of the size so we are looking to what we can do to store this data, taking into account raid space useage if we were to use an external. We would like to keep storage solutions WITHIN the company, ie not outsourcing for the actual storage space (similar to cloud storage ect).

I've considered the following:

- having a 60TB system with a Raid 0 to act as the backup
--OR--
-using a 30TB system then using Disks to back up the data

The issues and questions I have with each of these is that the raid would require our constant monitoring (as loosing the fisrt section of the raid jepardises the second ect) and the problem I can see with using disks is that the data coming from the proposed system will be coming in extremely quickly (60,000 x-ray images a day approx) so would the disk be able to keep up?

Does anyone have any suggestions for alternative solutions or any products we could use to do this? like I said before, cost is not an issue.

Many thanks! x

More about : solution storing 30tb data

a b G Storage
August 13, 2014 8:15:20 AM

How often do you need access to the older data and how long does it need to take to retrieve it if you need it? You could do something like a 6-10TB RAID array with a tape library backup system that holds 16 LTO5 or LTO6 tapes which are about 1.5TB to 2.5TB a piece (uncompressed).

Other simple options are just a RAID box that has enough space for your recent data and then use a hard drive dock and hard drives to store your data that you aren't using.

You could also do something like a dell power vault system if you need all of the data randomly accessible at all times. You can pretty much expand those out with large sata drives and expand out as much space as you need.

You could look at these too: https://www.backblaze.com/blog/180tb-of-good-vibrations...

m
1
l
August 13, 2014 8:17:26 AM

You will need 2 places to store this for a real backup, one on site and one off site. I recommend looking into getting a 60TB NAS (network attached storage) onsite and one offsite at a proper host who can really backup your data in case of a catastrophe.
m
0
l
Related resources
a c 130 G Storage
August 18, 2014 8:57:21 AM

You would want a storage with at least a 25% size overhead of what the estimate is. Using RAID 0 is NOT a good idea as that is not really used for data backup or recovery, it's just for speed and being able to use drives in one volume. Maybe you mean RAID 1, and even that is not that good in your case because that will slow things down some.

Amount of images copied does not much matter, it's the size of them that matters. Also make sure you are running the thing on a 1 gig network between the machines gathering the files and the storage locations.

You seem to be going at this from the point of a home user instead of an enterprise. Storage and servers require constans monitoring no matter what, if you are not doing that, you are not doing half your job.

If cost is not an issue, look for the cost of a solid state SAN or NAS, next you should check for one using SAS drives, some of which run at 15k RPM and are pretty speedy and are designed or enterprise use.

I've worked in a hospital for years supporting all sorts of imaging systems, althogh that was mainly for real-time image capture and retrieveal, not for older files. The big thing is to get a good fast network connection between all the systems. At 100mb our radiologists were complaining about lag in the system.
m
0
l
August 18, 2014 1:29:24 PM

As Traciatim said, the "best" solution really depends on answers to questions like how fast and how often you need to access your data. If all you need is straight archival, then you can probably just use tape. If you need a computing system to access your data quickly at all times you can look at a storage server.

DAS storage sounds like it would be enough for you. If you find a 20-40 bay box and fill it up with 1-2TB you'll cover the capacity you need. But it also depends on what RAID configuration you go with. hang-the-9 is right RAID 0 isn't a good idea because there is no redundancy there. And if your data is extremely important/sensitive you would want to go with at least a RAID 5 option so that if one drive fails your system will continue to run. And if you decide you need a more robust option like RAID 10 you'll need a larger system to go with it as half of your available space will be allocated to backup.

If cost is really no issue you might consider going with a networked storage option but I'm not sure you need any of the advanced features. DAS will have better throughput/lower latency which might benefit you more.
m
0
l
a b G Storage
August 21, 2014 10:34:24 AM

Guys,

It is obvious she needs corporate support from a well-known player - like Dell, HP, IBM, etc. 'Cause the moment it goes down, she needs to get it up again quickly. So whether that's buying a new server with a lot of local storage, or buying a SAN (or close to it, like Dell's lower-end PowerVault), it needs to come from an established vendor with a 4-hr response time.

I think I would be fine with buying a Dell PowerVault NX3200, and loading it out with as much storage into as it will handle natively (12 x 4TB). If you feel comfortable enough, you could even get a Dell PowerEdge 720xd with 24 x 3.5" bays and load it out yourself with high-capacity SATA drives (maybe the Seagate Enterprise Capacity series).

Some other items already mentioned were spot-on - definitely plan for more storage than the initial specs call for, and get some big pipes to push/pull that data (like an Intel quad-port GB adapter or a 10GB adapter along with a GB onboard for mgmt). As your backup, you can always go the easy route & just buy another of the initial unit identically configured.
m
0
l
a b G Storage
August 22, 2014 7:01:34 AM

Enterprise NAS storage, such as the FAS2554 from Netapp is also a solution. 10GE connectivity, built in NFS, CIFS, iSCSI. Get the dual controller HA version for availability. Easy to upgrade for space or performance. Use a tape solution for low cost off site disaster recovery.
m
0
l
!