Inside Intel's Secret Overclocking Lab: The Tools and Team Pushing CPUs to New Limits
We get an exclusive tour of the facility where Intel pushes chips to their absolute limit.
We regularly do large scale CPU testing in our own labs, particularly when we spin up CPU reviews with upwards of 15 chips in a test pool, so we know that selecting the right equipment, and having a lot of it, is imperative. That makes the OC lab tour all the more interesting, as we had the opportunity to poke around Intel's labs and compare notes.
And, of course, there's the eternal question that has spawned perhaps millions of heated forum exchanges: What's the best TIM (thermal interface material)? Intel probably wouldn't wade into that debate, but we asked the company what it uses in its own lab, and it turns out the company uses multiple types of TIM, largely dependent upon the testing it is doing.
Intel's lab has a $2,200 "tub" of Vince "Kingpin" Lucido's KPX, so it's obviously one of the go-to solutions, along with plenty of Thermal Grizzly's Kryonaut and Conductonaut. The company also uses an Intel-designed formula that is made by the third-party company Shin-Etsu. The blend isn't available to the public, but it doesn't seem to be the lab favorite and was mentioned as an afterthought.
Surprisingly, but not really too surprisingly given that this author thinks they're great coolers, Intel uses Corsair H115i coolers for testing in its lab, and there are plenty of them interspersed throughout. We also use these same coolers for CPU testing, and like Intel, we've found them to be very durable and hold up well to constant processor re-mountings. Intel uses the H115i for standard water-cooled overclocking testing but steps up to custom loops for overclocking HEDT models.
Intel's building has its own cool water supply for its various labs, but the overclocking lab uses a more powerful water chiller of its own. The chiller distributes water throughout the lab via two large tubes that extend along both sides of test benches, which you can see behind the monitors. Lab technicians can then simply plug into these chilled water supply lines at multiple stations inside the lab.
Interestingly, the overclocking lab's chiller is cooled by the building's cold water supply, meaning that there are two separate cooling loops in the lab. That allows the chiller, which creates its own waste heat, to be cooled via the central cooled water supply instead of exhausing that heat into the lab.
The company also has a piece of gear named The Medusa that we would absolutely kill for. The unit, pictured above, is a custom-made Peltier cooler that allows the company to keep a processor at a set temperature regardless of load. Intel designed this thermo-electric cooler and has used it worldwide in its facilities for roughly 15 years.
Stay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
The unit has a Peltier thermocouple that, once attached, will keep a processor at the specified temperature (say, 60C) under all conditions. This unit is also cooled via the building's cool water supply. Among many other uses, this type of cooling solution comes in handy for testing systems that may not be easily cooled due to a lack of conventional cooler mounts. For instance, it was used extensively for Hades Canyon testing.
If you're swapping motherboards constantly, a solid test bench is a must. Again, like us, Intel uses the Open Benchtable (OBT) as it's go-to mount for motherboards.
After years of searching for the best open-air test bench, even going back to the heady days of Danger Den torture racks, we can attest that the OBT is the best open-bench option for testing CPUs. You mount motherboards with a peg-like system, so there are no fasteners required when you swap out gear: You just pop the motherboard on and off when needed, but it remains firmly in place without fasteners.
The OBT also folds flat and can be thrown in a suitcase for travel, which is handy for Intel because it's technicians have to set up for demos and often run tests in hotel rooms and remote locations. It's also handy if you're a CPU reviewer because we often test in hotel rooms to meet NDA deadlines while covering trade shows or events (you'd be surprised how often that happens).
We can also see one of Intel's many thermal imaging units that it uses to spot hot spots and diagnose various issues with motherboards.
We counted four oscilloscopes in the lab, but this model stands out. The Tektronix MSO 70404C mixed signal oscilloscope is the most powerful model in the lab and runs about $68,000. This scope can cover up to roughly 8 GHz, which allows the engineers to examine and debug really fast interfaces, like the overclocked memory bus. The rest of the scopes are a bit more pedestrian ($20,000 range).
Interestingly, this model runs Windows, so Intel's security team had to work it over and install software to keep things tidy from a security perspective.
Intel makes plenty of alterations and modifications to motherboards to further its overclocking pursuits, both to its own internal designs and those from third-party motherboard vendors. As such, the lab has a rework station and an expert technician that can fix "just about anything." That also comes in handy during the company's overclocking workshops with motherboard vendors.
- PAGE 1: The Overclocking Lab
- PAGE 2: The Beginnings and Mission of Intel's Overclocking Lab
- PAGE 3: Pouring LN2, the OSHA Way
- PAGE 4: TIM, Coolers, The Medusa, and Other Intel Lab Gear
- PAGE 5: Validation Boards and Overclocking Bootcamps
- PAGE 6: VRM Supercooling, PCH Swapping, and Internal Tools
- PAGE 7: 'Safe' Overclocking Voltages and Techniques
- PAGE 8: Is Overclocking Dead?
MORE: Best CPUs
MORE: Intel & AMD Processor Hierarchy
MORE: All CPUs Content
Current page: TIM, Coolers, The Medusa, and Other Intel Lab Gear
Prev Page Pouring LN2, the OSHA Way Next Page Validation Boards and Overclocking BootcampsPaul Alcorn is the Managing Editor: News and Emerging Tech for Tom's Hardware US. He also writes news and reviews on CPUs, storage, and enterprise hardware.
-
PaulAlcorn Dark Lord of Tech said:Can you get the AMD tour? Would love to see that.
I'll jump on a plane the second it is offered :) -
bit_user @PaulAlcorn , thanks for the awesome piece!Reply
I'm still making my way through it, but wanted to draw special attention to this bit:
the engineers told us they feel perfectly fine running thier Coffee Lake chips at home at 1.4V with conventional cooling, which is higher than the 1.35V we typically recommend as the 'safe' ceiling in our reviews. For Skylake-X, the team says they run their personal machines anywhere from 1.4V to 1.425V if they can keep it cool enough, with the latter portion of the statement being strongly emphasized.
Thanks for that!
At home, the lab engineers consider a load temperature above 80C to be a red alert, meaning that's the no-fly zone, but temps that remain steady in the mid-70’s are considered safe. The team also strongly recommends using adaptive voltage targets for overclocking and leaving C-States enabled. Not to mention using AVX offsets to keep temperatures in check during AVX-heavy workloads. -
StewartHH Some one should comparison between different vendors die size like Intel 10nm vs AMD 7nm to see if there is actually performance gain. I would use per-core speed and not taking multiple cores into account.Reply -
bit_user @PaulAlcorn , uh oh. Now that I just finished heaping praise, I've got a gripe. In the penultimate paragraph:Reply
... assures that the learnings lessons and advances made in the overclocking realm ...
I was saddened to see the "learnings" virus infecting your otherwise admirable writing.
I think "learnings" is one of those pseudo-jargon words that MBAs and other B-school types like to throw around, out of jealousy for practitioners of real professions. Everyone from auto mechanics to accountants, lawyers, and doctors needs jargon to adequately and efficiently express concepts and constructs central to their work. However, common sense pervades business to such a degree that I think they're embarrassed by how easily understandable it'd be, if they didn't inject some fake jargon to obscure the obvious. The resulting assault on the English language is disheartening, at best.
Yes, if you've ever heard of her, you probably guessed I'm a fan of Lucy Kellaway, former journalist of the Financial Times and BBC. Worth a read:
The 8 Lucy Kellaway rules for claptrap and the fundamental theorem of corporate BS
Lucy Kellaway’s dictionary of business jargon and corporate nonsense -
Gurg AMD CTO Mark Papermaster: "you can't rely on that frequency bump from every new semiconductor node." AMD's future outlook of very limited frequency bumps, performance increases only from more cores and expensive software modifications to use more cores.Reply
VersusIntel Ragland: "People who think this the end of the world for overclocking because our competitors' 7nm has very little headroom, that's not true. Intel is all about rock-solid reliability; our parts aren't going to fail...you can count on your part running at spec, so there's so much inherent margin that we will always have overclocking headroom...I think users will be happy with the margin we can offer in the future."
Ouch! Intel's Ragland really "punked" AMD's negative outlook.
PS Great fascinating article -
jiang-v Anyone knows how to made contact with them? cause I fould a big bug on 10th corex chip about adaptive mode overclockingReply
overclocking/comments/ehxa7cView: https://www.reddit.com/r/overclocking/comments/ehxa7c/big_bug_in_10th_core_x_vid_mechanism_worst_avx512/ -
nofanneeded In the past OC gave a huge difference , today we can easy hit 4.4 all cores without OC and this is more than enough for me.Reply
for me OCing is dead. and I dont care about missing 5 fps.
I put the price difference in a better GPU ... -
CompuTronix Outstanding article! Thank you, Paul! I would love to have been there. I have a few dozen questions that the Team may or may not have been allowed answered.Reply
However, like bit_user, I found it of particular interest that the Team was forthcoming regarding specific voltage and temperature values they're comfortable with running on their personal home rigs, which max out at 1.425 and 80°C. With respect to electomigation and longevity, every day in the forums we see many overclockers express their concerns over these very issues.
On their website, Silicon Lottery shows Historical Binning Statistics that include the Core voltages used to validate their overclocked 14 and 22nm processors. For 22nm the maximum is 1.360. For 14nm the maximum is 1.456. While Intel's warranty is 3 years, Silicon Lottery's warranty is 1 year, which suggests at least one reason for the voltage difference between Intel's Team and Silicon Lottery.
Here's a forgotten link to a revealing Tom's Hardware video interview of July, 2016, with Intel's Principal Engineer (Client Computing Group), Paul Zagacki, where BGTnJkuqlbo']Intel Discusses i7-4790K Core Temperatures and Overclocking. The video coincides with the formation of Intel's Overclocking Lab, also in 2016. In the video, Intel points out that overclocking abilities begin to "roll off" above 80°C, which agrees with the value the Team revealed in your article.
While Core temperatures, overclocking and Vcore are often highly controversial and hotly debated topics in at least the overclocking forums, the term "electromigration" is closely related to a much less known term, which is "Vt (Voltage threshold) Shift". With respect to voltage and temperature, the two terms describe the causes and effects of processor and transistor "degradation" at the atomic level.
In the Intel Temperature Guide, in Section 8 - Overclocking and Voltage, I created a table for Maximum Recommended Vcore per microarchitecture from 2006 to the present. For 22 and 14nm, those values are 1.300 and 1.400 respectively. I also created a graph showing the Degradation Curves for 22 and 14nm processors. The table and graph helps overclockers get a better perspective of the degradation and longevity issue:
Sparing our members and visiting readers the deep dive, Vt Shift basically represents the potential for permanent loss of normal transistor performance. Excessively high Core voltage drives excessively high current, power consumption and Core temperatures, all of which contribute to gradual Vt Shift over time. Core voltages that impose high Vt Shift values are not recommended. The 14nm curve suggest 1.425'ish is the practical limit, which also agrees with the value the Team revealed in your article. The curve also suggests that Silicon Lottery might be pushing the edge of the envelope a bit.
The concern here is that when novice overclockers casually glance around the computer tech forums, where conflicting and misleading numbers get flung around like gorilla poo in a cage, many don't realize through the fog of all the confusion that one size Vcore does not fit all. Aside from high Core temperatures, Vcore that might be reasonable for one microarchitecture can degrade another. So 22nm Haswell users now wanting to overclock their aging processors to keep up with today's games need to heed the degradation curves, which applies as well to 14nm Skylake and Kaby Lake users.
CT :sol: