Help with unique network storage requirement

Toggle sidebar Toggle sidebar
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510
I have some somewhat unique network storage requirements I am hoping someone can help advise me on.
I have a small LAN with about 10 users. Each week, we receive 3-4 hard drives full of data (about 1-2 TB each) that are produced on a separate network. We can control how these drives are formatted to best suit our needs (ie. to ext4 vs ntfs, etc), but we cannot directly link to that network. We need to be able to put this data on our LAN so everyone has access to these drives. We need access to each drive/data for about 2 months, then we are done with this data and it can be removed from the network. As soon as these drives are added to our network, they are processed for about 2 days. During that time, good I/O speed is very helpful. After that, speed is less important.
Doing this with a traditional NAS setup seems like a waste, since it necessitates copying all that data up to the NAS each week when I only need the data for a short period. What I really want to be able to do is just hang those drives off the network for two months, then remove them when we are done. I can’t seem to find a NAS that allows me to just serve an already populated drive. I’ve been looking at file servers, but they seem like overkill for my application – and my budget. If anyone if willing to help provide me some expertise/advice, I’d really appreciate it.

 
Solution
B
Sounds like you need a computer/server with hot swap drive bays and have drive pooling so you can add the drive to the pool with predefined network shares, and then just remove the drive from the pool before you eject it.

Once you "process" the data on the drive, do you make backups or do any transferring of data on the drive to your local systems? or do you simply open/modify files on the drive and leave it on the drive?

Is the other network always the same or is each disk from different places?
Would a VPN connection to the other network and just accessing the drives that way be more effective?

Might be more helpful if you included just a bit more information on where the drives are coming from and what the process is.
Sort by date Sort by votes
kanewolf

kanewolf

Titan
Moderator
May 29, 2013
37,269
3,260
156,790
How many total "temporary" drives do you need access to at once? Many NAS units have both USB3 and eSATA external interfaces. I would think a docking station type external adapter would be the best answer. Connected either via USB3 or eSATA.
 
Upvote 0 Downvote
B

boosted1g

Titan
Jun 12, 2013
14,846
8
74,965
Sounds like you need a computer/server with hot swap drive bays and have drive pooling so you can add the drive to the pool with predefined network shares, and then just remove the drive from the pool before you eject it.

Once you "process" the data on the drive, do you make backups or do any transferring of data on the drive to your local systems? or do you simply open/modify files on the drive and leave it on the drive?

Is the other network always the same or is each disk from different places?
Would a VPN connection to the other network and just accessing the drives that way be more effective?

Might be more helpful if you included just a bit more information on where the drives are coming from and what the process is.
 
Upvote 0 Downvote
Solution
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510


We’re working on about 20-30 drives at any one time.
My understanding is that, at least for the NASs I’ve read about, even using the USB/SATA ports, I couldn’t hang an already populated drive but would have to put a new formatted drive on it -- but I’d be glad to be wrong about that. Do you have experience with a NAS that will serve a pre-populated drive that way? That would be a great solution for me if there is a certain NAS will allow that and has enough expandability through those ports.
 
Upvote 0 Downvote
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510


When the drive is processed, some of the results are kept locally, but the original data is simply opened, used, and left on the drive.
I agree the VPN would be a better solution for the data transfer/usage, but that network is not owned by me and the owners are not open to another solution (other than the transfer of data to physical media).
The drives contain disk images. Once we receive them, we extract data from them (the I/O intensive process) and then review that data. After the extraction, access to the original HDDs is limited but necessary.
The server solution seems to be what I keep coming back to, but my knowledge in that area is extremely limited. Can you recommend a server (or type of server) I should be looking at, and/or resources I might use to learn more about this type of server? Thanks for your help.

 
Upvote 0 Downvote
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510


Scheduled outages would not be a problem.
 
Upvote 0 Downvote
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510


Most of them are coming from the same place. Ironically, most of them come from the same building complex. Thanks for the recommendation - that may be workable. If they are willing to connect the small NAS to their system to copy the info off, I could then just connect it to mine for processing. Thanks for the idea. I had never considered using small NASs instead of drives for the transfer.
 
Upvote 0 Downvote
D

DeadlyDays

Honorable
Mar 29, 2013
379
0
10,960
"but we cannot directly link to that network" - I don't understand specifically what you mean by that.

get a hotswap disk tray, basically an external tray with sata hard drive connections that you slide a hard drive into. This then plugs into a computer somehow depending on product. Some are firewire, usb, etc. You can get ones you just add into like a disk drive slot on a desktop.
So basically take the drive and plug it into a computer that is serving the network for however long you need access to it. Some residential routers will act as a file server and have usb/firewire ports for that purpose.

If you don't want the actual drives accessible on the network, then you are going to need to clone the disk to another disk using some sort of software, which you can do the same way into a machine you designate for that purpose then share the cloned image onto the network.

Maybe I don't understand what you are wanting though.

**I posted this late, before those other answers***
20-30 drives is a lot.
You may want to look for a fileserver with a LOT of hotswappable bays. Or build one yourself and find a hotswappable bay you can attach to sata directly, and add additional via PCI cards, and see how many you can fit onto a single mobo. You may be able to get quite a few.

Or have them transfer multiple images onto single higher density drives or like mentioned above already populate NAS systems.
 
Upvote 0 Downvote
B

boosted1g

Titan
Jun 12, 2013
14,846
8
74,965
There is hardware that can do what you need but this is beyond the scope of just directing you how to setup 2-4 hot swap drives in a traditional desktop tower. With proper hardware you can add/remove drives on the fly without having to do a complete power down of the system. The hardware required for this though is not an off the-shelf type product and will require custom piecing it together and assembly/configuration by a qualified processional; while I and others on this forum are fully qualified for this kind of setup it is well beyond the scope of giving you some quick guidelines over the internet.

To do hot-swap change out like that you will need a special case designed to hold that many drives, and raid cards/port multipliers for that many drives. You also will need an appropriate power supply for that since 30 drives spinning for the first time takes a lot of amps (roughly 1.5 amps per hard drive).

You can then either share each drive individually as a separate share or you can setup drive pooling so that it shows all the drives as 1 individual share.
Either way will require a a couple minutes simple configuration with each new drive loaded in.
 
Upvote 0 Downvote
kanewolf

kanewolf

Titan
Moderator
May 29, 2013
37,269
3,260
156,790


Think of a bank or other "secure" environment. Having ANY external network access may be a no-no. Air gap is how data is entered or removed from those type networks. Not every network is connected to the internet or even another network. There are very good reasons for standalone networks.
 
Upvote 0 Downvote
C

circodegraves

Reputable
Oct 9, 2015
6
0
4,510


Thanks for your help. I have been looking at a couple of server options. I think I'll have to go the file server route, as none of the NAS solutions I have seen seem to have the ability to add and serve a pre-populated drive. I have been looking at some of the Dell servers, but I am not familiar with Windows Server 2012. If I set up (or rather, get someone to help me set up) a server using an OS like WS 2012, I will be able to add and serve pre-populated drives, right? I only ask this because every tutorial or manual I look at always talks about adding extra drives as if they are clean drives that are just being added to expand space on the system.
 
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom