Concerns over CPU Benchmarks

Are you actually serious about a "grand marketing conspiracy created by the chip makers"?

Aside from that, why worry about benchmarking software at all?

Any decent benchmarker will not only use those package suites, but also throw in real-world benches from software that's designed to actually do something, not just pretend to.

So long as software like 3DSM and games are benchmarked, any influence on packages like BapCo, Sysmark, etc. are relatively unimportant.

What bugs me are review sites that intentionally run their favorite CPU using a fully-tweaked top-notch platform, and run the 'competition' on a less-than-optimal hardware platform that isn't tweaked even a tiny bit. Biasing benchmarks through hardware is a <i>lot</i> more of a problem than just having one or two benchmarking software suites <i>possibly</i> corrupted.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>

The biggest question is. Do you use anything in Sysmark? The information on Sysmark's website tells you every application that is tested in each test. If those applications are important to you, then you can rely on the score, if not, ignore it. Simple as that. I never care about the Commache 4 benchmarks. Why? I don't play it. On the other hand, the JK2 benchmarks are vital, even the deciding factor for me. Along with any Lightwave/Maya benchmarks. Why? I use those. What do you use?

"We are Microsoft, resistance is futile." - Bill Gates, 2015.

Considering how many games are based on various revisions of the Quake 3 engine, I'd imagine that an awful lot of gamers should be looking at those Q3 results, even if 500FPS is completely meaningless in and of itself. It gives a good indication of how newer games (with higher object and polygon counts) using the same engine will also perform.

And best of all, Q3 is also a multi-threaded app, so it even scales pretty well on dualie systems.   It might seem like a very outdated benchmark, but it's still actually very applicable.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>

Ok, Q3A has a few multithreaded parts but even John "I can optimize a game for dual CPU" Carmack can't change the nature of the program. Games just don't have enough parallelism to extract multiple threads. Although Carmack may have been able to find a few places that he could use 2 CPU's for, it's hardly mind-blowingly beneficial, if even at all.

"We are Microsoft, resistance is futile." - Bill Gates, 2015.

Slvr I've come to conclude, you are right about how programmers no longer know the meaning of optimize.

Looking at Commanche 4 is a dead on example. I have never seen such crappy developpment in graphics that would put any GPU useless to use CPU. I do not beleive one bit the game was well programmed. No way josé. I have seen many games with far better graphics, that were GPU intensive and could run on old CPUs much better. No, Nova Logic should be ashamed as hell, they did not make any optimizations, they made a CPU intensive game.
The fact a friggin 2.8GHZ lays waste of most of its MHZ (we're talking 2.8 Billions of possible operations done in a second) to yank out 63 FPS at most, is shameful, especially on a Ti4600(I am not bashing the P4 here, I am saying the company is making any CPU waste itself). I hope Nova Logic's GFX programmers got fired or got a salary cut!

--
When buying an AthlonXP, please make sure the bus is at 133MHZ, or you will get a lower speed!<P ID="edit"><FONT SIZE=-1><EM>Edited by Eden on 08/28/02 05:00 PM.</EM></FONT></P>

This was the first game I've played that made my P3/850 and Voodoo5 system look like a complete dog. That's when I made the decision to finish all my older games first before getting a new machine to play the DirectX 8 and newer games. However, I did think Comanche 4's graphics were pretty damn good. Not spectacular, but very good. And that's with a Voodoo5 which doesn't support all the bells and whistles that a GF3-class card does.

Ritesh

True enough, but it does not hold the GFX crown and in no way is it in a position to be known as something that can naturally hog a machine, but it did because of lousy programming.

I thought the whole point of GPUs was to stop companies from making the CPU stress itself, let it cool while the card does the job instead, and let some AI stuff be handled there. This way you could potentialy end with smarter enemy AI, or at least the system would not lag as much on slower CPUs. Good example is Max Payne, IIRC. The scale test by THG shows a loss of 6 FPS between a P3 800 and a P4 2.53GHZ using R9700! Now that is programming that is worth commending, because at least people with slower CPUs won't have to pay more.

But Commanche 4 sets a new low in computer graphic programming, and though JK II is CPU intensive, at least it outputs 100FPS in almost any GF3 card and above, and that is enough not to complain.

--
When buying an AthlonXP, please make sure the bus is at 133MHZ, or you will get a lower speed!

Because of those reviews that you despise (as do I), that's the reason why Vans shut down. That, along with threats, hate mail, and some threatining pohne calls, and what vans calls "expenses".

Eden, I'm with you all the way. The words <b>programmer</b> and <b>optimize</b> do not go together in the same sentence. There are so many bloated games and programs out there (eg. Windows XP, Comanche 4) that just put alot of nice hardware out of business. I mean, seriously, it doesn't even matter what video card you use for Comanche 4, it won't go higher than 50fps. JkII can go high becuase it's built on q3a engine, although a heavily modified version.

- - - - -
Montecito - successor to Madison, and one monster of a CPU.

"And best of all, Q3 is also a multi-threaded app, so it even scales pretty well on dualie systems.   It might seem like a very outdated benchmark, but it's still actually very applicable."

I was rather suprised when I read the performance tests for the new P4's versus the newest Athlon XP. The XP got totally creamed. Not even an overclocked 3000+ could stay anywhere NEAR the new P4. That was rather suprising to me anyway.

Knowledge is the key to understanding

(CNET) <A HREF="http://news.com.com/2100-1001-956079.html?tag=dd.ne.dht..." target="_new">AMD: Chip test was altered to favor Intel</A>

Care Factor = <b><font color=red>ZERO</font color=red></b>

like one of the previous posters said... only concentrate on the benchmarks done by real programs that you will use!

<b>My CPU cooler is so Massive it bends Space and Time!   </b>

Unfortunatly SysMark supposedly uses real-world apps.
Though people like Van claimed that there were filters inside the tests, were used several times in order to put the P4 in the lead. While Van is an AMD biased foe, I kind of beleive him because the lead is so significant now, per clock and at the highest speed, that it's hopeless to base any AXP purchase by that.

--
When buying an AthlonXP, please make sure the bus is at 133MHZ, or you will get a lower speed!

Have you been keeping up w/ what has been going on? All that Van stated was a rehash of what AMD filled him in on through email. While it may be true, Van had nothing to do w/ finding it out. He just took info given to him by the AMD PR machine, then made it sounds like his own discovery.

This sig runs too hot.

I don't care what he did, I also want to know what is going on in this program, because cheating can seriously alter and confuse users.
Photoshop is more than often an Athlon favored, so how is it so good for P4 in that test?
In anycase I hope the next Sysmark with AMD's relation to BAPCo will have fair benching by both Intel and AMD pitching in, with each being aware of the other's tests.

--
When buying an AthlonXP, please make sure the bus is at 133MHZ, or you will get a lower speed!

The whole point of benchmarking is to give you a basic idea as to how much of a performer your system really is.

Knowledge is the key to understanding

Yes and when you see a bench repeating a certain filter for a certain processor to win, to jack up the score so you'd think it is considerably better when it could be only 3% better than the competitor, is cheating and lies to people on how much better is a CPU. It can make a difference in making someone pay an extra 200$ for that CPU, which in reality is 3% and not a cheated score.

--
When buying an AthlonXP, please make sure the bus is at 133MHZ, or you will get a lower speed!

This was the topic of Tom's latest (or first, if you've been at this site for only about a year) editorial. Nice to see that he's still alive. But, what he said is that the difference is so small that it's hard to say if there is some programming differences, especially when the one program that AMD absolutely owned the P4 was made less of a determining factor in the final score. Now, one could charge that those changes from the previous BapCo to the current version might have been engineered by Intel to make AMD shine a little less brightly in that suite of tests, but it's AMDs fault for being one of the few big computer companies not on BapCo. They recently remedied that, but again, it doesn't help this year's version, and maybe it'll have no affect on future versions either.

Needless to say, it's a moot point, since it's an artifical benchmark that uses an artifical number at the end to indicate performance, much like 3DMark and PCMark. It doesn't really test anything useful, besides how quickly people will cry foul.

-SammyBoy