Editor's Corner: Getting Benchmarks Right

It's All In The Numbers

In Monday's first look at AMD’s Socket AM3 interface, we observed some interesting gaming results on Intel’s Core i7 920 versus the new Phenom IIs (and subsequently got called out on them). This, of course, after overclocking both micro-architectures in a previous story and comparing their respective performances.

Of course, in that most recent piece, we used AMD Radeon HD 4870 X2 graphics cards, and in this one, we employed a pair of GeForce GTX 280s. It turns out that graphics makes all of the difference in gaming--who would have guessed?

Moreover, there were other sites that published their own evaluations of the AM3 platform, yielding a second round of comparisons.

Curious as to why, exactly, we were seeing different results from some of the other publications out there and in response to requests for more data from our readers, I dedicated the past two days to hypothesizing possible causes and re-running our gaming tests, using Far Cry 2 as my indicator of choice. A sincere thank you to the folks who posited helpful information  and constructive suggestions in comparing data.  I tried to replicate as many of the other test scenarios from Monday's round of reviews as possible here.

Without further ado let’s get into some troubleshooting, benchmarking, and hypothesizing.

UPDATE: At the suggestion of several readers, I've re-run some of these tests (still in Far Cry 2) with Hyper-Threading disabled and then again with the Threading Optimizations in Nvidia's driver disabled completely. The results are as follows:

Though there is some difference here, just like in the case of the fresh installation of Windows, our performance questions remain unanswered for the most part. Hopefully, these serve to cross one more possibility off of the list of explanations as to why the GeForce GTX 280 is under-performing when paired to a Core i7 processor. Now, the editorial as it originally appeared:

Possibility #1: Benchmarking with power-saving features enabled was causing performance problems with Core i7.

After disabling EIST (and hence turning off Turbo mode), C1E, and the thermal monitoring function that could throttle the processor if it broke past its pre-programmed 100A/130W limits, it was clear that this combination of features affected performance, but just slightly. Certainly, they weren't to blame.

Possibility #2: At DDR3-1066 (the max ratio of Intel’s engineering samples), we were starving our platform for memory bandwidth.

Rather than simply clocking the memory bus up to 1,600 MHz by upping the Bclk, we cranked the reference setting up to 190 MHz, yielding a 3.8 GHz clock speed and 1,523 MHz memory bus. The boost helped a tad at 1920x1200 and a little less at 2560x1600, but in no way made up the difference between a Phenom II at its stock settings.  No go there, either, which is incredibly strange. Graphics bottleneck, anyone?

Possibility #3: The driver install was bad, and the GeForce GTX 280 was actually at fault.

Starting clean, with a fresh copy of Vista x64 and a fresh driver installation, we started over using Nvidia’s GeForce 181.22 build—the latest available, and the one we used in our story earlier in the week. The i7 setup scored a bit higher, but still failed to pass any of the Phenom II X4s, which would have been the result we were looking for to show Core i7 in the lead here. Still, no go.

Possibility #4: Our results are only representative of gaming on an Nvidia card, and those using AMD Radeon boards will see something different.

No kidding, right? Of course gaming on a Radeon is going to give you a different result—especially in a game. But we didn’t expect variance to this extreme. Suddenly, we’re on to something. The i7 920’s results shoot up to levels that truly trounce the Phenom II X4 machine armed with the GeForce GTX 280 (as it should, given the faster processor and graphics card). We'll dive into a lot more depth on this point on the next page.

Possibility #5: Ok, turn down the CPU clock, silly. You’re still running at 3.8 GHz.

The results scale lower, most notably at 1920x1200, where we’d expect a CPU to have a more profound impact on gaming performance (versus 2560x1600, at least). But the 4870 X2 still leads the X4s and X3s by a commanding margin, even with the 920 running at its stock 2.66 GHz. And look how much more scaling there is coming from the overclocked configuration above.

Possibility #6: Web site XYZ used a different driver version. Perhaps something happened between that version and the most recent update you used.

Stepping all the way back to October of last year, we installed Nvidia’s GeForce 180.43 package to test the difference between then and now. And, if anything, the GeForce GTX 280 only picks up performance given the more recent driver update. That’s not the problem, either.

Possibility #7: Your card is hosed.

Fair enough, we have more than enough cards here to at least try swapping that out. We didn’t suspect the processor, memory, or motherboard of being defective—after all, the Core i7 920 at 2.66 GHz served up compelling results in all of our audio/video encoding apps. It was only the gaming scores that looked funny.

But that isn’t the issue either. A new card ranged from the same to slightly worse in Far Cry 2—certainly within a margin of error, to be sure.

Possibility #8: Maybe you guys use different settings that make more of an impact on the i7 920’s performance.

Now it feels like we’re reaching for straws. Nevertheless, we were willing to try the same Far Cry 2 batch using High settings instead of the Ultra High DirectX 10 configuration used to test initially. And again, there’s a significant increase, but it isn’t so substantial that Intel is able to usurp the fastest X4 under the load of Very High settings.