Sign in with
Sign up | Sign in
How Fast Is Thunderbolt, Really? Tom's Tests With 240 TB
By ,
1. A Wall Of Terabytes

When premium storage manufacturer LaCie, recently acquired by Seagate, approached our editor-in-chief Chris Angelini at CES, a few minutes of conversation left one big question: how much storage could we daisy chain onto a Thunderbolt connection, and how fast would it go before slamming into a bottleneck? How insane do you have to get in order to flood the PC world’s current flagship peripheral interface? LaCie volunteered to help us find out. Fortunately, I live about five minutes down the road from LaCie’s headquarters in Hillsboro, Oregon, and the company’s senior engineer, Mike Mihalik, twisted all the arms necessary to free up 240 TB of Thunderbolt storage to conduct the following experiments.

2. About Thunderbolt

One could describe Thunderbolt as an extension cord for the PCI Express bus. The interface is electrically identical to DisplayPort. However, whereas DisplayPort can only realize 5.4 Gb/s of throughput per Thunderbolt lane, storage devices can achieve up to 10 Gb/s. Systems manifest four Thunderbolt lanes as two full duplex pathways, with each lane offering that 10 Gb/s of bandwidth. Apple was the first to debut Thunderbolt on its MacBook Pro in 2011. PC vendors have been slower to follow suit, although Lenovo, Acer, Asus, and HP have all publicly committed to the technology. DIY motherboard buyers can find Thunderboard support from Asus, ASRock, Gigabyte, and MSI. If you want a true deep-dive on Thunderbolt, check out Everything You Need To Know About Thunderbolt.

3. Why The Mac?

On Mihalik’s recommendation, we tested using a MacBook Pro with Retina Display since he had extensive experience with the two benchmarks we're using, Disk Speed Test and AJA System Test. Our original preference, Iometer, does exist for Mac, but Mihalik has found it to be unreliable. As shown here, we have both Thunderbolt ports connected. Apple’s Disk Utility can create and manage RAID arrays through either or both ports. Because the tool tracks individual drives rather than their position in an enclosure, it’s easy to switch from using both ports to daisy chaining off of one port and measuring the impact on performance.

4. Different Drives

As you’ll see, we really went into this examination looking to answer two questions: 1) How much storage you can pile onto one or two Thunderbolt ports? and 2) What is the maximum real-world storage performance Thunderbolt can offer? We tested plenty of configurations along the way to our answers, from a single USB 3.0-based drive to the previously-sighted wall of 5big enclosures. Shown here is a pair of LaCie Little Big Disk Thunderbolt drives, each containing two 512 GB SSDs.

5. The Back Side Of 240 TB

While our 240 TB experiment is off the charts on geek sweetness, Mihalik was careful to offer caution. Would someone ever want to have a 240 TB RAID 0 volume comprised of 60 hard drives in real life? No. If anything happened to any one of the 60 drives, the entire volume would go down. So while our exercise here aims to prove a point, it’s mostly academic in its extremity. The higher your volume’s drive count, the higher the risk of failure. If this had been an actual storage array for a real application, we would have employed a RAID level with some fault tolerance, such as RAID 5 or RAID 6 (for double failure prevention). Naturally, this would have taken a toll on performance.

6. When Software Is Better

We asked Mike Mihalik if a system with a dedicated, hardware-based RAID controller would be more appropriate for a test such as this. He said no. These days, processors like the Core i7 in our MacBook Pro are so fast that they keep resource utilization on RAID 0 computation down to less than 2% on average—a negligible amount that has essentially no impact on other applications. Going with a discrete controller wouldn’t improve the performance we'll be reporting. Also, hardware RAID controllers tend to use proprietary formatting routines that dictate where data gets written on each drive in each enclosure. With OS-based RAID, such as that provided by Apple’s Disk Utility, users can pull out drives, switch their positions around, and the system will reassemble the array in seconds. This can’t be done in conventional hardware-based RAID configurations.

7. A Bug In The Soup?

In theory, up to six devices can be daisy chained from a single Thunderbolt port. With five 4 TB drives per enclosure and twelve enclosures total filling both ports, we should have had a single 240 TB volume running from our laptop. However, under Mac OS 10.7.5, Mihalik could get no more than six enclosures working, regardless of whether that was a 1 x 6 or 2x 3 configuration. It should work, but it doesn’t. With 10.8.2, we could see drives in all twelve enclosures, but only drives in eleven enclosures would mount. Specifically, we could see 59 of the 60 drives in play, but it takes all five drives to make one 5big enclosure work properly, so we effectively were left with 220 TB of active storage in one volume. Mihalik believes we may have stumbled across either a driver or OS bug, and he’s currently following up with Apple.

8. Internal Boot SSD

Let’s start at the beginning. This is our Disk Speed Test’s look at our internal 512 GB drive. Keep in mind that this shows the benchmark assessing the boot drive, so there will be an overhead penalty for the operating system and such. I personally like how Disk Speed Test provides at-a-glance notice on which video formats can be supported given the storage target’s throughput performance, although I wish it went up through 4K.

9. USB 3.0 Boot SSD

Not wanting to run from the older OS, we ran Mac OS 10.8.2 from a LaCie Rugged USB 3.0 Thunderbolt drive. This model uses a 120 GB SSD, and we ran it from the USB 3.0 interface. We include the drive’s performance here mostly as a point of reference, and to compare against the larger internal SSD. Mac OS makes booting from external storage remarkably easy, but you definitely pay a price on performance. Seeing an SSD knocked down to these levels is a shame, particularly from a 5 Gb/s interface.

10. One Thunderbolt Enclosure

Time to put a little work on our first Thunderbolt port. Here are the Disk Speed Test Numbers for one 5big enclosure, with all five 4 TB drives striped into a single RAID 0. In best-case situations, working at the outer platter tracks, a late model Barracuda XT can reach sequential speeds around 190 MB/s; innermost tracks cut this rate in about half. With five of these drives in an enclosure, subtracting some overhead loss and knowing that RAID 0 volumes don’t scale perfectly with each drive added, our results are in line with expectations. Note that one enclosure can meet the performance requirements for every listed video format save one (1080p59.94, 12-bit RGB 4:4:4 reads).

11. 2 x 5big, Separate Channels

As we piled on more storage, we brought in our second benchmark to corroborate Disk Speed Test. AJA System Test is a straightforward performance assessment tool for OS X with the added benefit of providing a throughput graph. Here, we’re testing a pair of filled 5big enclosures, with one enclosure on each Thunderbolt port. Our I/O graph looks fairly smooth, but AJA shows write throughput of 1,356 MB/s compared to Disk Speed Test’s 1,103 MB/s. We ran these benchmarks back to back, not concurrently. “What’s in the buffer, the calculation, arranging the data, accessing a different part of the disk; all of these can contribute to the 10% or so wiggle between each test,” says Mihalik.

12. 2 x 5big, Daisy Chained On One Channel

So how much of a hit do you take when daisy chaining from a single Thunderbolt port rather than spreading across both ports? Asked differently, why would you pay for two ports instead of just one? We took the exact same drive and test configuration from the previous page—the same RAID volume, in fact—unplugged the 5big from our MacBook’s second Thunderbolt port, and plugged it into the first 5big’s remaining Thunderbolt port. The result is a roughly 40% hit to performance. Oof! Nevertheless, this is still enough to pass every one of Disk Speed Test’s video format requirements.

13. 4 x 5big, Separate Channels

Let’s double down. We planted two 5bigs on each MacBook Thunderbolt port—40 TB of storage on each channel. If we keep Mihalik’s 10% wiggle room comment in mind, we get sobering news, indeed. Statistically, there is no difference between having one enclosure on each port and having two. One 5big per port effectively maxes out Thunderbolt’s peak throughput. This is a rather disheartening revelation. If you figure that each Thunderbolt channel specifies 10 Gb/s (1,250 MB/s), and we’re dealing with four channels, well...we’d hoped for more.

14. 4 x 5big, Daisy Chained On One Channel

And when we take that same 80 TB volume and daisy chain it onto a single Thunderbolt port? No difference. We see throughput in the 700 to 800 MB/s range, just as we did with two enclosures on one port. Adding more drives does nothing to help accelerate performance at this point.

15. 2 x Little Big Disk (SSD)

Lest you wonder if SSDs might somehow blow hard drives out of the water in this context, we grabbed a couple of LaCie Little Big Disks, each containing a pair of 512 GB SATA 6Gb/s drives. (LaCie specifies up to 635 MB/s for these.) You can see the photo of this config on page four. The good news is that we again see the benefit of balancing storage across both Thunderbolt ports, especially on reads. The bad news is that we’re not even getting close to beating our prior throughput numbers, even with four SSDs spread over both ports. Mihalik says that these results match his own testing at LaCie. A dual-port Thunderbolt system is unlikely to realize more than 1,400 MB/s today.

16. Behind The LaCienes

After hours of testing at the front of the building, Mike Mihalik and others took us on a tour around the LaCie facility. While LaCie is headquartered in Paris, the U.S. operation was founded in Tigard, Oregon in 1987, and the French operation was founded as D2 in Paris in 1989. While independent for most of its life, the company has changed hands twice, to Quantum and now Seagate, but the Paris location still hosts much of the company’s R&D work, as well as loads of validation testing. The Hillsboro location handles marketing and technical support, and hosts a manufacturing site. We were pleased to tour LaCie’s live phone support room. Yes, when you call U.S. support, you’re actually talking to someone in America.

17. Conclusion

So what have we accomplished? Well, first off, we may have stumbled onto an undocumented bug in Mac OS, which is pretty cool. More importantly, we demonstrated that Thunderbolt can live up to most of its promises and deliver some outrageously large storage volume capacities. In our minds, though, the biggest find here is identifying Thunderbolt’s performance ceiling. It’s quite clear to us that very high-bandwidth tasks, such as 4K video editing, will require either moderate SSD arrays or large hard drive arrays to be balanced across at least two Thunderbolt ports. And if those applications require more than 1,200 to 1,300 MB/s of throughput? Then you’re out of luck with dual-port Thunderbolt, and it’s better to know that now than find out in the middle of a critical job.