NAS too slow, will this be a better alternative?

[kaotik78]

I currently have a WD World book II drive setup to be raid 1. It has a cap of 10 MB/s which in my opinion is BS!

Is it possible if I was to get a WD Pro Drive with USB 2.0/firewire 400/800 that I could plug it into the server on my network and get transfer speeds faster than the puny 10 MB/s that WD has put on my lovely World Book II???

Im sorry but 10MB/s is just too damn slow for archiving large amounts of data and I need to find a viable alternative.

Any suggestions or thoughts on the idea are greately appreciated! Have a great night everyone!

 

[Madwand]

Yes, if you're talking about the transfer speed from the server to the directly attached USB2/Firewire drive. You should get somewhere around 30 MB/s with those interfaces.

You might do somewhat better with an eSATA interface. For this you'll probably need an add-on eSATA card and a drive enclosure which has an eSATA interface. You might even be able to re-use the existing drive in the WD enclosure provided you got an enclosure which matched the drive's native interface.

However, you have a 100 Mb/s LAN, so any remote access to this drive would be limited by the network speed, which would get you that "10 MB/s" effective at best in practice.

Here are some performance charts for external drives:

http://www.tomshardware.com/2006/08/24/seagate_500_gb_external_hard_drive_goes_esata/page5.html#write_transfer_rate

 

[kaotik78]

Damn it! I figured I was thinking out of the box on this one. My network is currently 100 mb/s. So even though the drive can go up to 480mbps transfer rate, when its plugged into a networked computer, and anyone trying to read/write to it will be limited to 10MB/s? What if I upgraded my network router and cards to gigabit ethernet, would I get higher transfer rates?

Im just trying to understand if gigabit ethernet can give you around 125MB/s in theory, why couldn't I transfer at around 60-80MB/s

I think I should just keep what I have and not go through the hassle of swapping the drive for the usb version to meddle with hooking it up to my w2k server and keep it available on the network??

Too many variables to consider, im gettin a headache

 

[Madwand]

There's theoretical speed, then there's the speed you see in practice. Even USB2 has 480 Mb/s = 60 MB/s speed in theory, but just look at the chart that I linked above and see if any of them come close to this speed. The practical speed is more important here, and that's around 30 MB/s.

Similarly, gigabit can go above 100 MB/s in theory, but in practice this is very rare. A number of ducks have to line up properly for this to happen, and this rarely happens in practice. Targeting 30 MB/s is a better idea in practice -- you're more likely to be able to get around this speed.

However, most consumer NAS boxes, and most probably the one you have don't hit such speeds even with good gigabit networks. Look at the Small Net Builder NAS charts (1 GB file size) numbers to see this. Such NAS boxes are typically powered by ducklings, not ducks, so of course they can't line up properly.

But, yes, if the basic ducks line up -- gigabit NICs on the computers, gigabit switch or gigabit router connecting them (definitely no wireless), CPUs not too bad, etc., a drive interface that can exceed that speed, then you would probably get around 30 MB/s network access to the backup drive.

 

[Iceblue]

Damn it! I figured I was thinking out of the box on this one. My network is currently 100 mb/s. So even though the drive can go up to 480mbps transfer rate, when its plugged into a networked computer, and anyone trying to read/write to it will be limited to 10MB/s? What if I upgraded my network router and cards to gigabit ethernet, would I get higher transfer rates?

Im just trying to understand if gigabit ethernet can give you around 125MB/s in theory, why couldn't I transfer at around 60-80MB/s

I think I should just keep what I have and not go through the hassle of swapping the drive for the usb version to meddle with hooking it up to my w2k server and keep it available on the network??

Too many variables to consider, im gettin a headache

You need to start with the hard drive specs and work back from there. No matter how you connect the drive, it will not be faster than it is.

The spec you want to look for on the hard drive is NOT the interface spec (PATA133, SATA 1.5, SATA 3.0 etc.) but the sustained throughput of the drive (sometimes expressed by the manufacturer as "buffer to disk sustained throughput" or similar words).

Here are two examples from Western Digital. The first is a desktop computer drive with a SATA 3.0 interface, 7200 rpm. The second is an "enterprise" drive, 10,000 rpm, SATA 1.5 interface.

WD3200YS 7200rpm Hard Drive, buffer to disk: 61 MB/sec.
WD1500ADFD 10K prm hard drive, buffer to disk: 84 MB/sec.

Notice that the drive with the slower interface is actually higher performance.

Nothing you do to either of these drives by way of packaging and connection will result in a higher throughput than the specified sustained buffer to disk throughput (for large files). You can have peak throughput that approaches the interface spec for small files that are already in the buffer.

Assuming you are sticking to desktop drives (much cheaper), that 61MB/sec translates to 488 Mbps. This means that you would never reach "gigabit" throughput with this drive, but you could easily max out a 100 Mbps network interface and even a firewire / USB 2.0 interface.

A well designed 100 Mbps ethernet should see throughput in the range of 10 MB/sec maximum.

A well designed gigabit ethernet LAN should see throughput in the range of 100 MB/sec.

So, what you are looking for is either a direct-connect drive (SATA, SATA 3.0, USB 2.0, firewire) or upgrading your LAN to gigabit and making sure your NAS enclosure will actually perform in the 400+ Mbps range. But, short of spending big bucks for enterprise server class drives, you are not going to get true "gigabit" throughput from your hard drive.

 

[thepustule]

Bearing in mind also that it really requires new powerfull hardware to really push that gigabit network. I seem to remember W2K server being mentioned. I really believe you'll be hard pressed to do better than 25MB/sec with that kind of server. MAYBE if you have a PCI-Express computer, with PCI-E gigabit and PCI-E SATA, and if you leave the server alone so it's not doing much (that means no 3D-pipes screensaver on the server), and if you make sure the gig card and SATA don't share the same IRQ, then you might be able to hit 60MB/sec. But for that I'd recommend using Linux/samba or Windows Server 2003.

You won't be taxing your server's CPU much here - mostly it's I/O capabilities.

 

[riser]

Everyone added in good information...

But not a single person mentioned your RAID-1 is slowing you down too. RAID 1 = mirroring, will be slower because both disks are writing the same information.

The fastest RAID configuration would be RAID-0 - disk striping, where data would be written faster than a single drive because its spanning across 2 drives. Failure is an issue with this because if one drive fails, you lose your data.

Mirroring, as you are doing, is the slowest RAID configuration, if I recall off the top of my head.. it only mirrors a drive, whereas most others stripe to increase read/write.

You should switch to either a disk striping set over a mirroring to increase your speed.. Hardware also factors into your limitations.

 

[Darth_Indy]

Good comment on RAID-1 being slower. It makes total sense.

- If reduntant backups are very important to you (that's why you're using mirroring right), why not have separate backup tasks to each drive. If you your goal is to have archival backup you could set up jobs to run at different times or on alternate days. This option would give you a most current and less current backup (depeding on when you set up the tasks).

- On the other hand, if your goal is to have the most current files backed up always, why not set up your backup SW to shadow your host drive to the NAS and set up the NAS as RAID-0?

- Don't forget the other information here either, it's good advice.

 

[thepustule]

Everyone added in good information...

But not a single person mentioned your RAID-1 is slowing you down too. RAID 1 = mirroring, will be slower because both disks are writing the same information.

.....

Mirroring, as you are doing, is the slowest RAID configuration, if I recall off the top of my head.. it only mirrors a drive, whereas most others stripe to increase read/write.

My experience is that it is not quite so bad as all that. RAID1 is definitely slower than RAID0 for writing only because you are writing two copies of everything. However, RAID1 is the same speed as RAID0 for reading if it is done properly, because the data being read will come alternatively from one mirror, and the other mirror, which speeds up read speed in the same way RAID0 does. (Although if you observe carefully, you will see that some cheap RAID devices, such as those from SansDigital, do not do this. On theirs, all RAID1 reading just comes from one drive - dumb! Other, more professional RAID1 implementations, such as IBM ServeRAID or the Linux Software RAID1, distribute reads between the two mirrors and get this nice speed increase.)

RAID5 is way slower than even RAID1 because it must algorithmically calculate parity information for all data written. Reading, however, is blazing fast on RAID5.

These days, however, hard drives fail so quickly and so regularly, that you've gotta be just plum crazy to actually use RAID0. So, in my opinion, the best alternative, unless you have piles of $$$ for RAID5, is still good old RAID1.

Bear in mind also that there are a few research articles out there about the increased risks of failure with RAID5 (to summarize, if you have lots of disks, there is a bigger chance that at least one of them will fail. And to make it worse, when you replace the failed drive and the array is rebuilding, it is busier than ever, at exactly the time when it is not fault-tolerant. This increases the risk of catastrophic failure even more because the disks are working extra hard to rebuild the array. The bottom line recommendation is to use RAID6 instead)

Food for thought...

 

[thepustule]

- If reduntant backups are very important to you (that's why you're using mirroring right), why not have separate backup tasks to each drive. If you your goal is to have archival backup you could set up jobs to run at different times or on alternate days. This option would give you a most current and less current backup (depeding on when you set up the tasks).

- On the other hand, if your goal is to have the most current files backed up always, why not set up your backup SW to shadow your host drive to the NAS and set up the NAS as RAID-0?

That would work, but there's one consideration that must be addessed. With all the data we are collecting these days (8-11megapixel pictures, movies, home movies, PVR recordings, digital music, downloaded ISOs, etc) we are starting to have LOTS of data hanging around. This means extra big backups. It also means TIME. If you store your data and OS on a redundant array (RAID1 or RAID5), in addition to proper backups, you have a much reduced time investment carrying out a recovery.

Normally, you will lose a disk in your array. If it is redundant, just replace, resync, and carry on. If it is not redundant, you will have to re-install your precious OS and hardware drivers, just to get the restore software running and accesing the backup location. THEN you will have to wait for AGES while all that precious data restores. It can literally take days, depending on the size of your accumulated data. And if you have family members (especially of the "significant other" variety), who have an outstretched machete at your neck ready to swing away if you lose 10 years of precious memories stored in digital family photos, you understand the stress)

Much better to use a redundant array and be up and running faster. That does NOT mean, however, that you can get away without doing backups - it's just that those backups should be a "last ditch effort" when all other, much easier alternatives (such as array rebuilding) have failed. This kind of scenario is much less likely, because arrays work quite well these days.

To summarize, I recommend a dual-tiered approach, including BOTH proper fault-tolerant arrays (RAID1,5,or6) AND proper backups. Lost data is a nasty thing. Believe me - I know. Once you've been bitten, you are much more motivated to protect those bits more carefully.

 

[Darth_Indy]

Good point...

I boils down to how much risk do you want to take on?

I may be screwed up on this but with my NAS I backup all data areas for every system on my home network and don't use RAID. The NAS also keeps one previous copy for each file it backs up. So I have a minimum of two copies for all files (the original on the PC and a backup on the NAS). That said, for very important files such as family pictures I backup the files to DVD almost monthly and I leave copies with my brother (my work laptop is also backup up on a separate system there).

Some paranoia is important. I haven't decided the best way to handle raw video files (before editing), but stand alone USB drives are getting really, really cheap. That may be the way for me to go.