A Wall Of Terabytes
When premium storage manufacturer LaCie, recently acquired by Seagate, approached our editor-in-chief Chris Angelini at CES, a few minutes of conversation left one big question: how much storage could we daisy chain onto a Thunderbolt connection, and how fast would it go before slamming into a bottleneck? How insane do you have to get in order to flood the PC world’s current flagship peripheral interface? LaCie volunteered to help us find out. Fortunately, I live about five minutes down the road from LaCie’s headquarters in Hillsboro, Oregon, and the company’s senior engineer, Mike Mihalik, twisted all the arms necessary to free up 240 TB of Thunderbolt storage to conduct the following experiments.
About Thunderbolt
One could describe Thunderbolt as an extension cord for the PCI Express bus. The interface is electrically identical to DisplayPort. However, whereas DisplayPort can only realize 5.4 Gb/s of throughput per Thunderbolt lane, storage devices can achieve up to 10 Gb/s. Systems manifest four Thunderbolt lanes as two full duplex pathways, with each lane offering that 10 Gb/s of bandwidth. Apple was the first to debut Thunderbolt on its MacBook Pro in 2011. PC vendors have been slower to follow suit, although Lenovo, Acer, Asus, and HP have all publicly committed to the technology. DIY motherboard buyers can find Thunderboard support from Asus, ASRock, Gigabyte, and MSI. If you want a true deep-dive on Thunderbolt, check out Everything You Need To Know About Thunderbolt.
Why The Mac?
On Mihalik’s recommendation, we tested using a MacBook Pro with Retina Display since he had extensive experience with the two benchmarks we're using, Disk Speed Test and AJA System Test. Our original preference, Iometer, does exist for Mac, but Mihalik has found it to be unreliable. As shown here, we have both Thunderbolt ports connected. Apple’s Disk Utility can create and manage RAID arrays through either or both ports. Because the tool tracks individual drives rather than their position in an enclosure, it’s easy to switch from using both ports to daisy chaining off of one port and measuring the impact on performance.
Different Drives
As you’ll see, we really went into this examination looking to answer two questions: 1) How much storage you can pile onto one or two Thunderbolt ports? and 2) What is the maximum real-world storage performance Thunderbolt can offer? We tested plenty of configurations along the way to our answers, from a single USB 3.0-based drive to the previously-sighted wall of 5big enclosures. Shown here is a pair of LaCie Little Big Disk Thunderbolt drives, each containing two 512 GB SSDs.
The Back Side Of 240 TB
While our 240 TB experiment is off the charts on geek sweetness, Mihalik was careful to offer caution. Would someone ever want to have a 240 TB RAID 0 volume comprised of 60 hard drives in real life? No. If anything happened to any one of the 60 drives, the entire volume would go down. So while our exercise here aims to prove a point, it’s mostly academic in its extremity. The higher your volume’s drive count, the higher the risk of failure. If this had been an actual storage array for a real application, we would have employed a RAID level with some fault tolerance, such as RAID 5 or RAID 6 (for double failure prevention). Naturally, this would have taken a toll on performance.
When Software Is Better
We asked Mike Mihalik if a system with a dedicated, hardware-based RAID controller would be more appropriate for a test such as this. He said no. These days, processors like the Core i7 in our MacBook Pro are so fast that they keep resource utilization on RAID 0 computation down to less than 2% on average—a negligible amount that has essentially no impact on other applications. Going with a discrete controller wouldn’t improve the performance we'll be reporting. Also, hardware RAID controllers tend to use proprietary formatting routines that dictate where data gets written on each drive in each enclosure. With OS-based RAID, such as that provided by Apple’s Disk Utility, users can pull out drives, switch their positions around, and the system will reassemble the array in seconds. This can’t be done in conventional hardware-based RAID configurations.
A Bug In The Soup?
In theory, up to six devices can be daisy chained from a single Thunderbolt port. With five 4 TB drives per enclosure and twelve enclosures total filling both ports, we should have had a single 240 TB volume running from our laptop. However, under Mac OS 10.7.5, Mihalik could get no more than six enclosures working, regardless of whether that was a 1 x 6 or 2x 3 configuration. It should work, but it doesn’t. With 10.8.2, we could see drives in all twelve enclosures, but only drives in eleven enclosures would mount. Specifically, we could see 59 of the 60 drives in play, but it takes all five drives to make one 5big enclosure work properly, so we effectively were left with 220 TB of active storage in one volume. Mihalik believes we may have stumbled across either a driver or OS bug, and he’s currently following up with Apple.
Internal Boot SSD
Let’s start at the beginning. This is our Disk Speed Test’s look at our internal 512 GB drive. Keep in mind that this shows the benchmark assessing the boot drive, so there will be an overhead penalty for the operating system and such. I personally like how Disk Speed Test provides at-a-glance notice on which video formats can be supported given the storage target’s throughput performance, although I wish it went up through 4K.
USB 3.0 Boot SSD
Not wanting to run from the older OS, we ran Mac OS 10.8.2 from a LaCie Rugged USB 3.0 Thunderbolt drive. This model uses a 120 GB SSD, and we ran it from the USB 3.0 interface. We include the drive’s performance here mostly as a point of reference, and to compare against the larger internal SSD. Mac OS makes booting from external storage remarkably easy, but you definitely pay a price on performance. Seeing an SSD knocked down to these levels is a shame, particularly from a 5 Gb/s interface.
One Thunderbolt Enclosure
Time to put a little work on our first Thunderbolt port. Here are the Disk Speed Test Numbers for one 5big enclosure, with all five 4 TB drives striped into a single RAID 0. In best-case situations, working at the outer platter tracks, a late model Barracuda XT can reach sequential speeds around 190 MB/s; innermost tracks cut this rate in about half. With five of these drives in an enclosure, subtracting some overhead loss and knowing that RAID 0 volumes don’t scale perfectly with each drive added, our results are in line with expectations. Note that one enclosure can meet the performance requirements for every listed video format save one (1080p59.94, 12-bit RGB 4:4:4 reads).
2 x 5big, Separate Channels
As we piled on more storage, we brought in our second benchmark to corroborate Disk Speed Test. AJA System Test is a straightforward performance assessment tool for OS X with the added benefit of providing a throughput graph. Here, we’re testing a pair of filled 5big enclosures, with one enclosure on each Thunderbolt port. Our I/O graph looks fairly smooth, but AJA shows write throughput of 1,356 MB/s compared to Disk Speed Test’s 1,103 MB/s. We ran these benchmarks back to back, not concurrently. “What’s in the buffer, the calculation, arranging the data, accessing a different part of the disk; all of these can contribute to the 10% or so wiggle between each test,” says Mihalik.
