Sign in with
Sign up | Sign in
Your question

CPU rating systems - how should it be made?

Tags:
Last response: in CPUs
Share
August 22, 2002 6:25:23 PM

I see lots of comments about the rating system vs. Mhz number here, however do anyone have an idea of how the rating system should be?

I am assuming that there are a lot of people here that should have far more knowledge than me about this issue, however, here are some of my thoughts:

Problem 1), since we are talking of the CPU, the rating system must go to the CPU alone, not a complete system.
Problem 2), "performance" can't be measured unless we know the exact variables and parameters for what makes a specific "performance rate".
Problem 3) The processor does not have only one task, and will therefore likely have different rates dependent upon the task in question.
Problem 4) Cache memory and other within the processor this may further generate problems calculating the processors true processing power.

- A unit may be offloaded for certain operations making those variables and parameters less important, however the CPU rating system cannot take this into account as it is beyond the scope of the processor.

Old and current systems
- Mhz. Not good indicator as performance depends on how many instructions can be made for each clock cycle, as well as
- Instructions per second. May depend upon the instruction set (A reduced instruction set would probably do every instruction faster, but may need more instructions to complete an operation)
- Flops (floating point instructions per second). Depends on presission(?), and floating points are not used that much by all users.

So any ideas for how the current cpu rating system can be replaced?
August 22, 2002 7:00:29 PM

I would like processors to be rated like softwares. E.G.--

P4 Willamete should be = P4
P4 Northwood 400 FSB should be = P4.1
P4 Northwood 533 FSB should be = P4.1.1

Athlon Classic should be = Athlon
Athlon T-bird 200 FSB should be = Athlon V2
Athlon T-bird 266 FSB should be = Athlon V2.1
Athlon XP should be = Athlon V3
Athlon XP T-bred should be = Athlon V3.0.1
Next Athlon core Barton should be = Athlon V4

For naming processor of same core with different frequency I like-

P4 Willamette 1.4 GHz = P4 model 1
P4 Willamette 1.5 GHz = P4 model 2
P4 Willamette 1.6 GHz = P4 model 3

For Athlon XP

Athlon XP 1500+ = Athlon V3 model 1
Athlon XP 1600+ = Athlon V3 model 2
Athlon XP 1700+ = Athlon V3 model 3

This kind of processor naming and rating system will remove all kind of confusion to rate two cpus of same or different class.

There should be a law to force all cpu manufacturers to maintain this kind of specific rule for naming and rating their cpus.
August 22, 2002 7:13:53 PM

My thought is simply by replacing one meaningless number (MHz) with two more meaningful numbers: IOPS and FLOPS.

An industry (or government, depending on who gets annoyed enough to do it first) designed and regulated 'benchmarking' OS gets developed. It'll probably only run on x86 at first, though in the future that'll hopefully change. All that the OS does is run benchmarks upon bootup. One measures a range of standardized integer operations and spits out an IOPS number, and another measures a range of standardized floating point operations and spits out a FLOPS number.

Then all CPUs are forced to be packaged with these numbers in plain view and further all producers are required to provide this information to the public. (Much like FDA food labelling.)

The industry (or government) provides an agency which runs these benchmarks. This agency's results are the only results which can be displayed on the packaging.

It isn't a perfect system (since it doesn't cover after-market packaging such as complete computer systems, nor does it cover deceptive marketting labels), but at least this way the information can't be hidden from consumers who actually want to know. And with a seperate score for IOPS and FLOPS, it'll ensure that people can relatively guage how well the processor will behave in the applications that they use.

The important key is to design the benchmarking OS to use a combination of IOPS and FLOPS that acurately portray overall functionality <i>without</i> unique optimizations (hardware or software) such as SSE, SSE2, 3DNow!, etc. If a manufacturer wants optimized benchmarking results to show the benefit of XXXXX, they'll have to do the work themselves and put out that data in addition to the regulated benchmark results.

P.S. Obviously, the next question that begs answering is what to do about platform adjustments to benchmark results such as memory bandwidth, etc. The answer is simply that the regulatory benchmarks use the absolute best hardware available for the processor at the time. If better hardware comes out later, too bad.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>
Related resources
August 22, 2002 7:21:18 PM

So then how do we know how a Pentium V4.2 model 1 compares to a Celeron V3.2 model 3 to an Athlon V3.2 model 2 compares to a C V3.0 model 4 to a Crusoe V1.0 model 3?

All that these model numbers do is completely obfuscate their performance. So how do we compare the models to each other?

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>
August 22, 2002 7:21:48 PM

believe it or not, but apple, before amd, was the first to start crying out that mhz isn't everything. when amd and intel were busy battling each other for the top "speed" processors last year or two, apple simply gave up all hope since they knew their motorola processors weren't gonna ramp up in speed as fast as intel/amd's. amd sure didn't mind promoting their "faster" proc is better than intel's.

now that amd has following the footsteps of apple. why not just forget the whole PR thing and just have a cryptic designation: A4. like amd's A4 vs intel's P4 vs apple's G4.
no one cares about ghz, pr, bah!.. if you're interested in performance (as should everyone) look at the benchmarks. does it matter if its and axp 19000+ or a p4 19GHz anymore.. i hope educated people look at the benchmarks and not the pr or clock speed. if so, cpu rating systems is a non-issue and this is a meaningless discussion.

<font color=green> there's more to life than increasing its speed -Ghandi</font color=green>
August 22, 2002 7:24:09 PM

i'd buy a AXP MACARONI and P4 CHEESE (if they call it that) as long as it performes better than anything.. doesn't matter what marketing monkeys do.. the benchmarks is what i look at. hrm.. making me hungry now.

<font color=green> there's more to life than increasing its speed -Ghandi</font color=green>
August 22, 2002 7:56:06 PM

Tech-savvy people don't fall in the trap of model numbering of cpu manufacturers. They know which cpu shines at which sector. So comparing different processors is easy for them.

Actually performance comparing rating system is needed for Average Joe people who clearly need to be informed in short words.

There was a day when P2 was crashing K6-2 or Athlon crashing P3 and some of it's variations crashing willy P4 and NW P4 at 400 MHz FSB.

Now when comparing AXP and 533 FSB P4, no cpu wins clearly in all segment. So MHz or performance rating can't truly indicates one cpu's lead over other cpu. Average Joe people don't benefit from this kind of rating system. Today the only the way to know about a processor's capabilities are having good knowledge. Most of Averege Joe people don't have time or interest about knowing deeply processors.

A rating system that rates a processor of same class is ideal. Average Joe people don't try to compare between Athlon and P4. They usually buy P4 (mostly with i845 + PC133 SDRAM) without knowing how it performs compared to other cpu's.

Quote:
All that these model numbers do is completely obfuscate their performance. So how do we compare the models to each other?

Forget about Average Joe buyers. They will buy knowing this processor is P4.1 model 1 or P4.1 model 2

<b> There is no true performance indicating rating system now and will never be any in future </b>
August 22, 2002 7:59:02 PM

Quote:
now that amd has following the footsteps of apple. why not just forget the whole PR thing and just have a cryptic designation: A4. like amd's A4 vs intel's P4 vs apple's G4.

Actually, AMD tried that too. Athlon marking went Athlon, AthlonB, AthlonC, <b>Athlon4</b>, Athlon XP. Appearantly their marketting staff never learned their Alphabet song from Kermit. I guess it really isn't easy being green.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>
August 22, 2002 8:00:13 PM

Quote:
Problem 1), since we are talking of the CPU, the rating system must go to the CPU alone, not a complete system.

This removes cross platform CPU rating (Intel/Athlon) because you can't use the same platform. Now, the only thing you could do is have a base reference point to start from with each system, but you won't quite get a true benchmark without being able to use the same platform.

Quote:
Problem 2), "performance" can't be measured unless we know the exact variables and parameters for what makes a specific "performance rate".

Benchmarks are built to do that, but few people use benchmarks in everyday use of a PC. Even those who do, different situations in an application my happen at different rates of frequencies with different uses. This also doesn't take into account that some processors are more efficient at certain tasks.

Quote:
Problem 3) The processor does not have only one task, and will therefore likely have different rates dependent upon the task in question.

See previous issue.

Quote:
Problem 4) Cache memory and other within the processor this may further generate problems calculating the processors true processing power.

Cache and other enhancements help in real world apps, so why would we care how a CPU performs without it? The current P4 would be seriously crippled with no L1 or L2 cache, but who would actualy run a computer like that?

Quote:
- Instructions per second. May depend upon the instruction set (A reduced instruction set would probably do every instruction faster, but may need more instructions to complete an operation)

That depends on the process, but it's more a buzz word to say how efficient the CPU is. Doesn't mean that it's the better CPU, as MHZ effects how usefull this number is.

There realy is no good way to accurately rate a CPU. Too many faactors. The best way to get close to what you need is to benchmark it with apps taht you use, but that still may not be 100% accurate.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 22, 2002 8:09:30 PM

I'd only like that if I could get a firmware update for my CPU to increase the speed.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 22, 2002 8:13:29 PM

This firmware update should be overclocking! Unlike firmware updating, it will not be supported by cpu manufacturer.
August 22, 2002 9:25:34 PM

I surely agree with you that cache memory and so on is necesary and we want to see the total performance, however, I see this as one of the main obstacles for making a uniform CPU test.

A example to show what I mean:

Is the CPU i.e. 2.53G P4 running with 400 Mhz memory faster than a 2.4G P4 running at 533 Mhz?

The answer is clearly yes, however we know from the benchmarks that the 2.4G system defeats the 2.53 system.
(System benchmark vs. CPU benchmark).

So Due to the inclusion of cache memory on the CPU we are no longer talking about a pure Central Processing Unit anymore. Surely - the Through-output is what interest us more, but that is like a system test vs. a CPU test on a smaller level. This makes it really hard to to make an overall rating of one system vs. another, thus it represent a problem for a true benchmark.
August 22, 2002 9:44:53 PM

I believe that for the mass market, we don't want to focus too much on the processor at all. There should be an individual system rating that take into account everything from chipset to the signals leaving to the screen.

- This could be given with several independent benchmarks, like graphics/multimedia, office applications, and so on, just as made here. (Renamed to system benchmark instead of CPU benchmark).

For the CPU however, I am not sure if instructions per second, or flops is enough(?) When AMD first came around, they got big because few floating point calculations where necessary, and they channeled resources to more important operations instead. Had it not been for the new floating point intensive games that came around, we would likely not focused so much on these numbers now.
I am quite sure Instructions per second is a strong candidate, however, the actual performance depend upon the instructions the processor actually can perform. A system needing generally three instructions for doing a general task would be far better than one needing to split it up into ten even at twice the number of instructions per second.
- So we might probably will need to find out what instructions or operations to do the benchmark on.
August 22, 2002 11:50:30 PM

No set of theoretical number would be accurate at all. Be it operations per second, clockrate, etc. The average consumer won't really care about which number is more valid, they'll only care about the higher number. Even if we did come up with some kind of overall benchmark, companies would design their systems to be more favorable towards that benchmark, not to mention use their influence to skew that benchmark.
Simply put, there is no way to "educate" the average idiot because the average idiot doesn't want to be educated. They won't even care that the higher number isn't the higher performance, they just like to think so. Why is the enthusiaste community even worrying about such things? Leave marketing to the marketing department.

"We are Microsoft, resistance is futile." - Bill Gates, 2015.
August 23, 2002 12:37:20 AM

i think the enthusiasts community is worried because of such things is because (hmm redundant there) even though marketing don't affect us, it affect tons of people who are not computer savvy.

ok let's put it this way imgod2u. you heard of crest and colgate toothpaste? tell me which is better and why. NOW (i'm gonna turn this completely on you), tell me WHY you don't buy the store brand toothpaste? it's cheaper.. the price/performance is alot better than crest or colgate! you yourself are victim of being "average idiot" yourself.

let people buy what they want as long as they're happy with it sheesh. if i want to pay $5 for my toothpaste instead of $0.99, let me be.

ok back to the posting. (lol) talking about PR rating huh? well, it's BS and we all hate it. amd marketing is sweating bullets, i'm sure they hate it too, but they have do use something.. imagine a crooked used car salesman trying to make a living, the more he scams people, they more his family is better off... i'd say amd, you should abandone the tbird bs, the comparison to p4 bs, and double.. hrm, quadruple your PR+ number to go against p4 if you like.

if amd is sucessful in "educating" the public to look at true performance, it shouldn't matter what the PR is against whatever Tera-hertz p4.

<font color=green> there's more to life than increasing its speed -Ghandi</font color=green>
August 23, 2002 12:52:44 AM

The point is that marketing will always be marketing. Why is the enthusiaste community worried about how "accurate" it is? No marketing can be accurate and still be effective because no one else who is competing with you will be "fair and accurate". Why do you even care how they market it? It's working isn't it? AMD got some market share with their PR system, does it matter if it's lying or not? You can't "tell the truth" with the average idiot because the average idiot only cares about flashy numbers, period. Such things should not concern the enthusiaste community because, simply put, PR doesn't affect them. You know a PR "2600+" doesn't neccessarily mean it'll perform exactly like a 2.6 GHz P4, what more do you want?

"We are Microsoft, resistance is futile." - Bill Gates, 2015.
August 23, 2002 3:40:50 PM

The cache memory runs at the same speed as the processor. You're not talking about cache memory, but system memory.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 23, 2002 4:10:17 PM

Quote:
The point is that marketing will always be marketing. Why is the enthusiaste community worried about how "accurate" it is? No marketing can be accurate and still be effective because no one else who is competing with you will be "fair and accurate".

Which is exactly why the government should be stepping in. You can't buy a load of bread or a gallon of milk without a label somewhere stating what a serving size is, how many calories the product has, how many calories of that are from fat, etc., etc. And even in fast food joints where they somehow get around needing to post that on everything, they're still required by law to provide that information should you ask.

Why should CPUs be any different? Force the manufacturers to use a government-regulated <i>set</i> of benchmarks and put that information on the product's packaging for retail sales, and make the information available should anyone ever ask for OEM sales.

Easy, simple, and to the point. Then even if the average idiot actually wants to know, they don't have to worry about a tech site like www.amdmb.com or www.apple.com giving biased benchmark results, because they'll have government-regulated benchmark results that are easy to find.

Quote:
Why do you even care how they market it? It's working isn't it? AMD got some market share with their PR system, does it matter if it's lying or not?

Because obviously the whole system is getting far out of hand and only the truely obsessed can even find the real truth behind everyone's lies and manipulations. Leaning the simple and basic facts of a product should <b>never</b> be so difficult.

That and I'm getting kind of tired of helping the typical idiots over and over and over again with answering the exact same questions.

In the Age of Information, correct information should <i>not</i> be this difficult to find. If there was no such thing as an idealist we'd all just be living in chaos. Someone's got to care, or no one else will. Heh heh.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>
August 23, 2002 4:20:42 PM

Quote:
NOW (i'm gonna turn this completely on you), tell me WHY you don't buy the store brand toothpaste? it's cheaper.. the price/performance is alot better than crest or colgate! you yourself are victim of being "average idiot" yourself.

I don't even but any of those toothpastes. I but a 'natural' toothpaste because I don't like the thought of swishing a known poison around soft tissues (our gums) to be absorbed into the blood stream and places where food could pick up unrinsed particles of the poison and transport it into my digestive system.

Sure, the human body is designed to filter poisons, but why force it to do so when we have other choices...

My point? Not everyone cares about the price/performance ratio. Some people have different needs than that, and those needs should be addressed at least to a minimal extent as well.

<pre><A HREF="http://www.nuklearpower.com/comic/186.htm" target="_new"><font color=red>It's all relative...</font color=red></A></pre><p>
August 23, 2002 6:31:35 PM

Quote:
My point? Not everyone cares about the price/performance ratio.

exactly... all i'm saying is that if amd fanboys are so concerned with price/performance then they should promote "natural" or "cheap" "generic" (eww) toothpaste too. they do just as good a job don't they? so in the discussion on price/performance they'd rock wouldn't they? .. i thought i had a point here.. oh well.

<font color=green> there's more to life than increasing its speed -Ghandi</font color=green>
August 23, 2002 6:49:58 PM

Quote:
You can't buy a load of bread{/quote]
You must eat a lot of bread.

The issue with food is different simply because of that. We consume food. We don't consume CPUs (Well, we try not to). People don't have alergic reactions to CPUs (Why clothes have material labels).

Now the closest thing I could think of to a CPU as far as specs would be Stereo equipment like a reciever/Amp. Amps have ratings of performance at different levels (And the lower end ones actualy advertize the biggest number as opposed to the most accurate number. Most 150W Sony amps are actualy only 65W amps). This is not well regulated, and you have to trust the manufacturer to put correct information or even supply it. And those statistics are much easier to analize fairly and accurately.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 23, 2002 6:57:32 PM

Price/performance people are looking for is different in different markets. For example, your "Stereotypical" retired person is a penny pincher who drives a Caddy. Doesn't make sense, and isn't consistant, but humans rarely do or are.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 23, 2002 7:00:59 PM

I actually think AMD is failing to educate the users with the PR number... The reason: The PR rating is supposed to show "True Mhz" and tell us that we can convert everything into a Mhz scale and compare...

What we need is a system that for example go entirely away from Mhz and for example look at total through-output.

The problem is that the current total through-output depends upon all the other components in the system, as well as the specific tasks to perform.

Could a CPU be tested in a controlled environment - not connected to a typical computer, but one a system being able to feed it and capturing the outout so all bottlenecks remain within the CPU?

We should probably stop calling the current tests CPU benchmarks and start calling it system benchmark instead.
August 23, 2002 7:10:47 PM

Post deleted by kandresen
August 23, 2002 7:11:36 PM

I might be wrong here, but I thought L2 cache run at a different speed than the processor, and is part of todays CPU's. (- If so, then they are adding to the complexity as we now have another bottleneck to take into account).
August 23, 2002 7:15:12 PM

No, the L2 runs at the same speed as the CPU and is integrated into the CPU itself. So Cache is part of the CPU as much as anything else.

The Boogie Knights: Saving beautiful monsters from ravoning princesses since 1983.
August 23, 2002 7:25:06 PM

ok, thank you for updating me!
a b à CPUs
August 23, 2002 11:19:54 PM

LOL, you could do it like Apache software:
P5 Pentium would be P1.1
P54C Pentium would be P1.2
P55C would be P1.12 (lol)
Pentium Pro would be P2.1
Pentium II would be P2.11
Pentium III would be P2.21
Pentium III B would be P2.121
Pentium III Coppermine would be P2.2
Pentium III EB would be P2.211

LOL, the numbering system asumes you know where all the 0's go!

<font color=blue>You're posting in a forum with class. It may be third class, but it's still class!</font color=blue>
!