Sign in with
Sign up | Sign in

Now We're Getting Somewhere...

Editor's Corner: Getting Benchmarks Right
By

So where’s the beef?

After a full day of breaking hardware down, reconfiguring software, downloading drivers, and endless loops of Far Cry 2, our results are still looking good. We ran them and re-ran them before Socket AM3 was launched, and we ran multiple iterations of them again in response to reader commentary on that story itself. AMD’s Phenom IIs are indeed running faster than the i7 920 in a number of our gaming tests where Nvidia's GeForce GTX 280 handles graphics.

Far Cry 2
1920x1200, no AA
2560x1600, no AA
Phenom II X4 940 and GeForce GTX 280 1 GB  (Numbers From The Launch)63.06
48.09
Phenom II X4 940 and Radeon HD 4870 X2  (Numbers Added)68.95
65.30
Core i7 920 and GeForce GTX 280 1 GB (Numbers From The Launch)
53.23
41.83
Core i7 920 and Radeon HD 4870 X2 (Numbers Added)85.87
74.85


Surely, the most interesting benchmark results come here at the end, though. Compare the scores of the two configurations with GeForce GTX 280s (Phenom II comes out ahead) with the scores of the two Radeon HD 4870 X2-equipped setups (Core i7 920 comes out in front). There are the scores everyone was expecting.

Pinpointing The Bottleneck

Could it be, then, that at 1920x1200 and 2560x1600, today's processors are so powerful that even the once-mighty GeForce GTX 280 runs headlong into a brick wall? Is the Phenom II simply more efficient than Core i7 when there's a major graphics bottleneck being presented? In order to answer that set of questions, I again built up my X58-based Core i7 platform to test with more GPU muscle: a pair of GeForce GTX 280s in SLI. Unfortunately, I couldn't run an equivalent setup using the 790FX-based AM2 or AM3 motherboards, and we don't have any SLI-capable AM2 platforms in-house, so we'll have to be content connecting a few dots with additional testing here instead.

Far Cry 2
1920x1200, no AA
2560x1600, no AA
Core i7 920 @ 2.66 GHz with one GeForce GTX 280 (Numbers From The Launch)
53.23
41.83
Core i7 920 @ 2.66 GHz with two GeForce GTX 280s (New Results)
86.68
70.80


Pre-testing Hypothesis: It's hard to draw parallels to our Core i7: 4-Way CrossFire, 3-Way SLI, Paradise? piece because we didn't test the Phenom X4 9950 with multiple GeForce GTX 280s there. We did test with Radeon HD 4870s, though, and noticed a curious case of reverse scaling as AMD was trying to get its optimizations in place for the game. We're hoping that a pair of GTX 280s demonstrates clear scaling here so that we're not left with a more severe issue centering on Nvidia's drivers. At least solid scaling would support the guess that an individual GTX 280 is unable to keep up with these modern processors.

Post-testing Analysis: Those numbers are much more indicative. Notice how the two GTX 280s are able to get much closer to the single Radeon HD 4870 X2 (but not beat it, strangely enough, at 2560x1600). We're just about ready to call this one a wrap. But first, a few more sets of numbers pushing resolutions and graphics settings down to see if we can accomplish the equivalent of running an AMD-based SLI platform at these more demanding benchmark options.

Far Cry 2, High Settings
1280x1024, no AA
1680x1050, no AA
Core i7 920 @ 2.66 GHz with one GeForce GTX 280 (New Data)
84.03
76.50
Phenom II X4 940 @ 3 GHz with one GeForce GTX 280 (New Data)
91.73
86.87


Pre-testing Hypothesis: If the hypothesis is correct, then we should see Intel's Core i7 920 assume its lead once the emphasis falls away from graphics muscle.

Post-testing Analysis: Indeed, it is not. Instead, the Phenom II maintains its advantage over Intel, even when you drop down to resolutions no gamer with this sort of hardware would really want to play at. If this is still a graphical bottleneck for the GeForce GTX 280, then our conclusion from the AM3 launch holds true, and the Phenom II really is faster in real-world gaming scenarios.

Far Cry 2, Low Settings
640x480, no AA
Core i7 920 @ 2.66 GHz with one GeForce GTX 280 (New Data)
179.01
Phenom II X4 940 @ 3 GHz with one GeForce GTX 280 (New Data)
124.69


Let's take it to an extreme, now. At 640x480 and ridiculously stupid-low settings, which make Far Cry 2 a synthetic metric for the most part, Core i7 finally takes its elusive lead. In other words, with absolutely all graphics load alleviated, the Core i7 starts replicating the sort of performance we saw in the A/V tests, handily trouncing the Phenom II.

Was it an Intel problem? An Nvidia problem? Simply Phenom II kicking ass and taking names in our gaming tests?

Our next order of business is to either pin this on Nvidia or vindicate the graphics vendor by running a single Radeon HD 4870 512 MB at 1920x1200, where we can expect it to get taxed pretty hard. We would have used a 1 GB card if there were any left around the lab. Leaving the 2560x1600 test out of this one should do the trick, though. 

Far Cry 2
1920x1200, no AA
Core i7 920 @ 2.66 GHz with one Radeon HD 4870 512 MB (New Data)
54.44
Phenom II X4 940 @ 3 GHz with one Radeon HD 4870 512 MB (New Data)
54.59


Pre-testing Hypothesis: If this is, in fact, a processor thing, the Radeons will demonstrate the same behavior as Nvidia's more powerful GeForce GTX 280, leaning in the direction of AMD's Phenom II X4 940.

Post-testing Analysis: An identical graphics bottleneck suggests that this is not the same behavior seen before. We're seeing a definite graphics-bound condition, where the GeForce GTX 280 is hitting its limit at different places depending on processor architecture. The implication is that Nvidia's drivers are not allowing the i7 to reach its full gaming potential.

Bottom Line

The bottom line here, first and foremost, is that all of the data generated and seen in the Socket AM3 launch piece was, in fact, right on the money.

The data suggests that, using an AMD Radeon-based graphics card, you'll likely see the scaling that many other sites have presented, with Intel's Core i7 besting the Phenom II right up to 2560x1600 (refer to the first chart on this page for proof there).

At 640x480--a largely synthetic measure of processor performance, the Core i7 rules the roost under the power of a GeForce GTX 280, too. But again, the graphics load here is minimal. Anything higher--even 1280x1024, another resolution you'd expect to be CPU-bound on these cutting-edge platforms--and Nvidia's card cannot translate the Core i7's microarchitecture into the same performance advantage, giving AMD's Phenom II-series chips the advantage seen in the AM3 story and in the two pages you've just read.

I want to make it a point to thank Nvidia--specifically Nick Stam--for working with me in covering every possible base as we explored where and how the GTX 280 was being affected. As soon as we get more depth from the company's driver team, I'll post an update here. But, for the time being, I hope that the testing invested here helps clarify any questions about the validity of our benchmark results in the Phenom II / Socket AM3 launch story.

Ask a Category Expert

Create a new thread in the Reviews comments forum about this subject

Example: Notebook, Android, SSD hard drive

Display all 96 comments.
This thread is closed for comments
Top Comments
  • 26 Hide
    dattimr , February 11, 2009 5:05 AM
    Nice one. Tom's is getting its act together again. Keep it up, guys.
  • 14 Hide
    Hamsterabed , February 11, 2009 5:51 AM
    How very odd, when i saw the benches i immediately thought there was a problem. Glad you guys made an article to explain and backup you numbers and i hope we get some answers. don't have another driver fail Nvidia...
Other Comments
  • 26 Hide
    dattimr , February 11, 2009 5:05 AM
    Nice one. Tom's is getting its act together again. Keep it up, guys.
  • 14 Hide
    Hamsterabed , February 11, 2009 5:51 AM
    How very odd, when i saw the benches i immediately thought there was a problem. Glad you guys made an article to explain and backup you numbers and i hope we get some answers. don't have another driver fail Nvidia...
  • 6 Hide
    Tindytim , February 11, 2009 6:53 AM
    Wow...

    Just wow.

    Right when I considering leaving this site forever for it's over Mac loving, Tom flashes me a glimmer of hope.
  • 4 Hide
    rdawise , February 11, 2009 7:04 AM
    Thank you Chris for this follow-up article..now where is kknd to argue....

    I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).

    I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.
  • -4 Hide
    rdawise , February 11, 2009 7:04 AM
    Thank you Chris for this follow-up article..now where is kknd to argue....

    I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).

    I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.
  • 0 Hide
    sohei , February 11, 2009 7:40 AM
    "I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons"

    wow 7-7-7-20? this is the performance...indeed
    P2 works with ddr2 great and you wary about timings
  • 2 Hide
    Anonymous , February 11, 2009 7:42 AM
    great article!!
    just a thought: what about previous generation of nvidia cards? could be this is a GTX 260/285/280/... problem. maybe you could try with one of 9xxx series.
  • 8 Hide
    StupidRabbit , February 11, 2009 8:10 AM
    awesome article.. only two pages long but it changes the way i look at the previous benchmarks. good to see you focus not only on the hardware itself but also on the benchmarks with a real sense of objectivity.. its what makes this site great.
  • 1 Hide
    cobra420 , February 11, 2009 9:11 AM
    so it looks like a gpu issue . why not try a gtx 295 ? or is that why you set the video so low ? now you found the issue theirs no need to try a different card ? ati sure did a good job on there 4870 series . nice job toms
  • 0 Hide
    Anonymous , February 11, 2009 11:27 AM
    Maybe Farcry optimized it more on ATI, maybe Intel is throwing sticks at the wheels of nVidia at the hardware level, maybe, maybe ... :S
    Why is Intel supporting multi-ATI config, but not multi-nVidia? Why doesn't Intel let nVidia use its Atom freely? Why, oh why?
    There are so many factors. I think if you replace Farcry with a synthetic test, there will be less unknowns. Just maybe :) 
  • 8 Hide
    jcknouse , February 11, 2009 11:31 AM
    Fantastic work, Chris. Simply awesome. I really, really, really enjoyed that analysis.

    Something I need to ask tho:

    Is the GTX280 a dual processor VDA, like the Radeon 4870X2? If not, wouldn't you expect significant gains of 2 GPUs over 1?

    Also...

    Wouldn't you expect to see even a small decline in performance (possibly miniscule/negligible...but still present) simply because running 2 individual VDAs in 2 PCI-E x16 2.0 slots requires work to be sorted between two physical devices (handling going over the SLi interface), whereas using the Radeon 4870x2 work is sent to and split on-card?

    I'd really be interested in seeing a performance difference between 2 4870s and 1 4870x2 with the same catalyst version, and latest firmware.

    Also, I am quite shocked at the nVidia suffering. I've been an nVidia customer for years. I am going to have to look at going ATI Crossfire if I build a new AM3 gaming platform later this year.

    This is one of the reasons I value Tom's so much...articles like this.

    Thank you again, Chris. This was invaluable to me both technically as well as a consumer looking for the best bang for my buck, even tho my budget isn't limited.
  • 1 Hide
    arkadi , February 11, 2009 11:40 AM
    to the point, simple, logical, grate job!
  • -7 Hide
    jeffunit , February 11, 2009 12:07 PM
    Why are you running games which stress graphics for a cpu review?
    Wny not run *cpu benchmarks*, which measure cpu performance for a cpu
    review?
  • 3 Hide
    Anonymous , February 11, 2009 12:07 PM
    I would love to see a full testing article of AMD cpu's with ATI and NVidia cards VS Intel CPU's with ATI and NVidia cards. Maybe the architecture of the ATI cards works better with Intel chips and Nvidia works better with AMD chips. Wouldn't that be ironic since ATI is AMD.
  • 2 Hide
    squatchman , February 11, 2009 12:15 PM
    Much better, it was probably a ton of work, but you see that people complain less with comprehensive results.

    To jcknouse: Yea, the 4870x2 is two GPUs on one board and it should be beating the GTX280. To jeffunit: Check the original article for the non-synthetic CPU benchmarks, or any other article for that matter.
  • 3 Hide
    bf2gameplaya , February 11, 2009 12:37 PM
    Finally some quantitative analysis from Tom's, with reasonable methodology and an interesting subject. Well done, you have educated me!

    Yet, I see that Far Cry 2 was the only title tested and I have no way of knowing if any of these conclusions carry over to any other title.

    But now I have the right questions to ask, Thanks Tom's!
  • 2 Hide
    jameskangster , February 11, 2009 1:06 PM
    Chris, thank you for the editorial. I really appreciate the fact that you guys try your best to respond and answer our nit-picking comments and questions. Now based on what I have observed from your articles, I have some suggestions that might help (or might not at all). What if the bottleneck is related to the motherboard and its memory timing setup?

    I did a quick comparison of your past setups for benchmarks (From this article, 2009-02-09 AMD AM3 article, 2009-01-07 Phenom II review article, 2008-11-03 i7 review article).

    Basically, you used the same hardware for i7 920 in articles 2009-01-07 and 2008-11-03
    Motherboard: Intel DX58SO Revision 403
    Memory: A-DATA DDR3-1600 2X2GB set to DDR3-1333 CL 7-7-7-20
    Video: MSI N280GTX-T2D1G-OC

    Also just as an external reference I used Anandtech's setup from this article
    http://www.anandtech.com/cpuchipsets/showdoc.aspx?i=3512&p=3. For those of you who hate Anandtech I apologize, but this article had the most comparable hardware setup:

    Motherboard: Intel DX58SO
    Memory: Qimonda DDR3-1066 4x1GB (7-7-7-20)
    Video card: eVGA GeForce GTX 280

    In these articles Intel's i7 920 2.66 GHz performance seemed to be dare I say better than Phenom II (although in one article Phenom II did not exist, but just looking at raw numbers) specifically related to gaming benchmarks. The interesting point here is that they all used Intel DX58SO motherboard, using 7-7-7-20 timing for the memory. The number of modules varied; however, it didn't seem to make a huge difference from what I have read so far.

    In the 2009-02-09 article and its subsequent editorial article Tom's Hardware used the following setup for i7.
    Motherboard: Asus Rampage II Extreme (X58/ICH10) LGA 1366
    Memory: Corsair Dominator DDR3-1600 8-8-8-24 @1.65V 3x2GB (caveat here Tom's hardware did try overcloking in the editorial article so that does vary)
    Video card: Nvidia GeForce GTX 280 1 GB

    Overall the video card chip model remained the same, the driver revisions were different but not siginifically; however Tom's used a different board and its memory setup was drastically different from its previous setups.

    It would be interesting to see the differences in performance comparing these setups (some of these setups might not be possible due to hardware/BIOS limitations, I didn't have time to look into that part):
    1. Rampage mobo with 8-8-8-24 memory timing vs Intel DX58SO with 8-8-8-24 memory timing
    2. Rampage mobo with 7-7-7-20 memory timing vs Intel DX58S0 with 7-7-7-20 memory timing
    3. Intel DX58SO with 8-8-8-24 memory timing vs Intel DX58SO with 7-7-7-20 memory timing
    4. Rampage mobo with 8-8-8-24 memory timing vs Rampage mobo with 7-7-7-20 memory timing

    Obviously I would use i7 920, the same video card and driver for above setups.

    I highly doubt that the memory timing would cause such a performance difference. My bet is on the motherboard.

    Also it would answer another question, or maybe obsfucate even more. Was it truly Nvidia's video card's fault that i7's potential was not translated into raw performance? or was it due to the motherboard, or due to memory setup? or both? or all three?

    I'm not expecting another editorial article for this, but it would be good to see get this straightened out. I'm really hesitant to place the blame on Nvidia for this yet.
  • 1 Hide
    Ho0d1um , February 11, 2009 1:06 PM
    Madis KalmeThere are so many factors. I think if you replace Farcry with a synthetic test, there will be less unknowns. Just maybe

    This article is testing real world results and not just number crunching
  • 0 Hide
    kknd1967 , February 11, 2009 1:21 PM
    This is simply a great analysis. thumb up for Chris' good work. I too originally suspected NV card + i7 behavior or X58 chipset in gaming. 2 other websites have i7+NV260 leading in CPU bounded low res test, but falling in GPU bounded high rest test significantly, which just does not make sense for a typical system behavior (usually one would expect leader still leads but by smaller margin in high res).

    What is interesting is now it seems best to use Intel i7 with AMD video card for gaming. So who is the loser? Nvidia...
  • 1 Hide
    jonyb222 , February 11, 2009 1:30 PM
    Quote:
    The data suggests that, using an AMD Radeon-based graphics card, you'll likely see the scaling that many other sites have presented, with Intel's Core i7 besting the Phenom II right up to 2560x1600 (refer to the first chart on this page for proof there).


    Quote:
    Nvidia's card cannot translate the Core i7's microarchitecture into the same performance advantage, giving AMD's Phenom II-series chips the advantage seen in the AM3 story and in the two pages you've just read.


    So, if I understood correctly, currently Nvidia cards/drivers are slowing down intel processors while AMD one can go full speed (which is why Phenom beats I7 in that case). And ATI cards let's both processors go fulll speed (which is why I7 beats Phenom in that case)

    Bummer for Intel/Nvidia, Horray for AMD/ATI?
Display more comments