How We Test Graphics Cards

Testing on an open bench requires careful planning, else it leads to misguided results. Most gamers have closed cases, affecting thermal, acoustic, and even performance results compared to exposed hardware. We're working to represent that more accurately.

It is impossible to deny the truth our readers have provided as feedback on multiple occasions. And we have long been aware of the shortcomings tied to testing graphics cards in open-air test beds, where the results we generate almost always differ from those collected in a closed case. Is there a solution that makes everyone happy?

Over the past six months, I developed a concept for our laboratory that allows us to combine the pleasant advantages of an open test setup (quick and easy access to hardware) with the changes our readers requested. The devil is in the details though, as a number of additional components are needed to accommodate our infrared and power consumption testing methodologies.

The next step was to get in touch with manufacturers at Computex 2016, where I found a partner in Lian Li. The company truly listened to what reviewers actually need.

The original PC-T70, shown above, is a classic open-air test bench afflicted by the same issues we were trying to avoid. Therefore, I drew up a modification that'd match our requirements pretty well.

Assembly and Reconstruction of Lian Li's PC-T70

In order to make this a closed case, there will be an optional set of side plates, a back plate designed to closely match standard PC cases (including an I/O shield), and an acrylic cover.

Each of the side plates is made of solid, black-colored steel and can carry up to two 120mm or 140mm fans. They're also equipped with a dust filter that can be attached using a magnetic lock. Although Lian Li typically favors aluminum construction, we specifically asked for steel because it's less expensive and more utilitarian.

To fully enclose the internals we also needed a cover. And since we wanted to look inside, the solution was naturally acrylic. This cover is perfectly suited to pop on and off via magnetic strips. I consciously decided against a hinge because it'd take up too much space with the case open.

From here on out, you won't be reading about components that we expect will be available to everyone. A lot still has to happen before we're able to fully utilize our equipment, necessitating a bit more customization.

For starters, there's our high-res Optris PI640 that can't easily take measurements inside of a closed case. Since we don't need the whole armada of fans, we have two additional side plates that can be attached instead to cover the side openings. Both parts are positioned on top of each other with a layer of special foil in between, which is permeable for infrared waves and whose transmittance we know.

This kills two birds with one stone because the IR radiation can escape, while warm air stays inside. After all, that was the whole point of our airtight exercise.

Since the new readings needed to be kept compatible with previous measurements, this setup employs similar hardware as our current system. Unfortunately, our old motherboard was damaged by electrostatic discharge, giving us a good reason to switch from a non-conductive test bench made from acrylic to a grounded case. And so MSI's X99S XPower AC is replaced by an MSI X99S XPower Gaming Titanium. The RAM also had to be switched out for four Corsair Vengeance DDR4-3200 modules. They operate at the same 2400 MT/s as the old ones, though. So, as far as performance is concerned, nothing changes.

We changed RAM modules because Crucial's Ballistix kit sports reflective heat spreaders. In some tests, they distort (if only marginally) the IR results due to their reflective surface.

Water Cooling

Air cooling isn't desirable for a number of reasons. First, a large tower-style heat sink would block too much of the IR camera's view. Second, in a very small case like this, the cooler would dissipate so much waste heat that mainstream graphics cards with lower TDPs would be thermally overpowered, affecting the measurement results. Water cooling helps us work around both challenges.

Our solution employs a classic open loop composed of individual components. The pump (an Alphacool Eispumpe VPP755) is decoupled and mounted in the lower part of the system, where it rests on foam. To the pump, we attach an Alphacool NexXxoS UT60 Full Copper 360mm radiator measuring 6cm wide and loaded up with three PWM-controlled be quiet! Silent Wings 2 fans in a push configuration. 

The reservoir is positioned on the side, where it's easily accessible. Through a tee fitting on the top, we're able to refill liquid at any time through a screw-on funnel.

Here's where the quick-release magnetic strips come into play, making it easy to get inside and install a water block on the graphics card, for example. You can even see in the images below where Lian Li drilled tubing openings for this exact purpose.

Fans, Sensors, and Other Components

Separately, I put together a mid-tower system inside of be quiet!'s Silent Base 800 using a fairly similar platform, logging the temperature over time of Asus' hot Radeon R9 290X StriX. The air, GPU, and motherboard temperature readings I collected served as a baseline to help configure the new machine's fan speeds.

The airflow of a standard PC case is simulated by a fan in the back for exhaust, plus a controllable fan on the opposite side of the case pulling in fresh air. That back fan spins at a fixed 700 RPM, while other one is dialed in to 800 RPM. The fan on the side can also be removed, its vacancy  filled with a cover. 

At the front right, you can see a small, aluminum heat sink. It is placed at the warmest spot inside of the closed case, and it houses a temperature sensor. More on that later. I also attached two small USB speakers on the floor behind the front, since nothing goes better with benchmarking than some music.

A modified 850W be quiet! Dark Power Pro 11 is hidden under the case's floor. Its modular cables had to be altered somewhat so they'd work with our oscilloscope's clamps, giving us the most stable voltage readings possible. The chassis itself serves as a path to ground, to which we made several connections.

Before we move on to our methodologies, let's review the speeds and feeds:

Test System
Hardware:Intel Core i7-5930K @ 4.2 GHz
MSI X99S XPower Gaming Titanium
Corsair Vengeance DDR4-3200 @ 2400 MT/s
1x 1TB Toshiba RD400 (M.2, System SSD)
2x 960GB Toshiba TR150 (Storage, Images)
be quiet! Dark Power Pro 11, 850W power supply
Windows 10 Pro (with all updates)
Water CoolingAlphacool Eispumpe VPP755
Alphacool NexXxoS UT60 Full Copper 360mm
Alphacool Cape Corp. Coolplex Pro 10 LT
Case:Lian Li PC-T70 with extension kit and modifications

MORE: Best Graphics Cards

MORE: Desktop GPU Performance Hierarchy Table

Create a new thread in the Reviews comments forum about this subject
This thread is closed for comments
10 comments
    Your comment
  • DookieDraws
    Now that looks like a lab! Interesting read, Igor.
    4
  • kwc813
    I like the article. That said, will they make THAT BENCH (or offshoot therof!) to sell? I've seen most out there, used more than a few and am tired of customizing. This would allow one to customize everything else and leave the platform just as it sits. Nice job!
    0
  • zthomas
    love the glass box.. that system is high art.. the pics made my computer pic file.. to be posted elsewheres..
    0
  • edlight
    I remember when there were comparisons of video quality and enhancements between ATI & nvidia, and results of an "HQV" DVD quality test for multiple cards. I've seen no mention that AMD, in their drivers since 7.5.1, have cut off detailed settings for video - you can't combine high sharpening and dynamic contrast, for instance. And they even now don't let you control the overall brightness, contrast, and gamma. They refer you to an MS thing in Win 7 where you set the brightness on your monitor. Well, ever monitor I've had set to 100% brightness still needed an adjustment at the video card to bring out the shadow detail.
    The enhancements are great - I run my R7 240 on the old drivers at 100% sharpening and dynamic contrast on, affect web video, etc. DVD's in Media Player Classic look like something much better, and 1080 looks 3D. It only works on certain players, though. It works for Flash mp4-AVC. I've always wondered if it would work on everything in Linux?
    I see that the cheapest nvidia with the latest enhancements is the GTX 1050. I'm wondering if it can sharpen the new YouTube? I have to use a Firefox plugin to force YouTube back to Flash to get the sharpening.
    I wish reviewers would get into that stuff, and for each card.
    I'd love it if reviewers of video cards would attend to these things.
    0
  • Lucky_SLS
    Looking forward to the updated performance stats Igor. nicely done BTW
    0
  • FormatC
    Anonymous said:
    I like the article. That said, will they make THAT BENCH (or offshoot therof!) to sell? I've seen most out there, used more than a few and am tired of customizing. This would allow one to customize everything else and leave the platform just as it sits. Nice job!


    I dicussed it with Lian Li and they will sell the core version of the T70 (without the cover) and an optional upgrade kit with all other parts to close it. This can help to minimize the financial risk for Lian Li and the costs on the customers side. Everybody can buy what he really wish and not verybody needs the full program. This closed "Real World" table was my idea and I hope, the industry and also reviewers will like it too.

    The lab here was built over the years, but I was every time not really satisfied with all my benchtables. The power consumption thing I started in 2013, the infrared measurement in 2014/15 - long time before other sites copied this. The audio lab was built with the help of a good friend to realize the room-in-room concept. The location is nearly perfect and I spent a lot of time and money to finish it. I'm an audio-freak and measured in Germany tons of speakers and headsets too. This was really successful because we have only HiFi magazines with a lot of phrases for the so-called Golden Ears or mostly very flat and useless reviews on websites without any measuring. I tried to get the right balance between theory and real life, combined with understandable data and conclusions.

    And who I am? I moved to media ( + product development/consultations) in 2012 and was working the half of my life in the German industry (the last position as lead programmer and for quality control). That also means, that I'm significant over 50 and also know the basics of production processes and a lot of "secrets" behind the scene. For reviews it is not only interesting, how a product performs and where are possible issues - it is every time very fascinating and exciting to find also solutions or workarounds (like EVGAs thermal pad mod) and to communicate with the manufacturers. I used and I'm using a lot of time to visit factories and headquarters in Asia to get even more contacs and sources.

    I worked in the last years mostly for Tom's Hardware Germany - but I'm really happy, that we found new translators to bring my content also to the US/UK site. So I'm here and you must endure this now ;)

    Quote:
    have to use a Firefox plugin to force YouTube back to Flash to get the sharpening.
    At first - thx for your suggestions. I will think about it. But especially Flash is a dead horse and it is not worth to spend more time for it. HTML5 is a better standard and YT is moving all content step by step. Maybe, we can make sometime a video special to take a closer look at all this problems. For HTPC builds it is not uninteresting :)
    0
  • zifn4b
    Dude! I use that wallpaper too. Excellent choice!
    0
  • Olle P
    Great idea! ... but:
    Quote:
    Air cooling isn't desirable for a number of reasons. First, a large tower-style heat sink would block too much of the IR camera's view.
    Possibly.
    Quote:
    Second, in a very small case like this, the cooler would dissipate so much waste heat that mainstream graphics cards with lower TDPs would be thermally overpowered, affecting the measurement results.
    I'd think the opposite way:
    * The heat generated by the CPU will be evacuated by the red fan. Shouldn't affect the graphics card one bit.
    * Some of the heat generated by the (high TDP) graphics card will impede the CPU cooling, which can affect CPU performance and thereby influence the test results. This is a viable part of real use and therefore a good thing to have in the test! It indirectly says if the graphics card will be more or less likely to require water cooling for the CPU to get the most performance out of it. (The main reason for having graphics coolers that blow as much of the heated air as possible directly out through the rear rather than just heating the case interior.)
    0
  • Unolocogringo
    Hi Igor,
    Just wanted to say that I really enjoy the depth and precision of your reviews.
    Ive been a member of Toms Hardware since 96, and have watched the progression of reviews evolve over the years.
    Your latest addition should make your reviews much more in line with consumer experience, and I applaud you for your effort and insight.
    Thanks
    Rick
    1
  • junkeymonkey
    why a 850w PSU ?? would it not be better to use a 550w seeing how there recommended here in the forums for a nice gaming rig ??

    if asked here at toms that 850w is way too much overkill .
    0