Vista: Benchmarking or Benchmarketing?

[pschmid]

Windows Vista features built-in benchmarking to assess system and component performance, by calculating so-called Windows Experience Indices (WEI). Can Microsoft really replace traditional benchmarking?

 

[proof]

They have replaced everything else...

 

[mpasternak]

someone forget to test AMD cpu's?

 

[robreto]

I installed Vista twice (as beta). I got two different sets of performance ratings on the exact same hardware.

The ratings were similar, but they were definantly different. I remember getting penalized bigtime for only having a modest (at the time) 6600GT 128MB video card.

 

[MickK]

I really would like to see the on-board graphics from Intel added to this for comparison.

We all know that MS fudged that program to make it look less embarrising for Intel. I'd love to see the actual comparison in black and white to see how much of an exception MS made to help Intel out.

 

[Newf]

Windows Vista features built-in benchmarking to assess system and component performance, by calculating so-called Windows Experience Indices (WEI). Can Microsoft really replace traditional benchmarking?

No.
The point is to ensure that a given user has enough computing horsepower to run Vista, and to identify where improvements are needed. Nobody in their right mind would use that as a serious performance benchmark. Would they?

 

[enewmen]

Can someone explain what those MS numbers mean?
Ok, 1 is lowest and 5 is hghest. You need X benchmark to play Y game requiring an X.
But what happends in a few years? The Playing tomorrows software on todays hardware will not work well.
Do you expect numbers will keep going up, 6,7,8,9, etc?
That way a '4' in 2007 will be the same as a '4' in 2010.

I underatand the Windows Experience Index us just a rough guess so the casual buyer won't need to read so much gobbledygook about "System Requirements:"

 

[Newf]

The numbers will go up over time, as new hardware comes out.

 

[enewmen]

The numbers will go up over time, as new hardware comes out.

Thanks. Now I can sleep more easily.

 

[bourgeoisdude]

Windows Vista features built-in benchmarking to assess system and component performance, by calculating so-called Windows Experience Indices (WEI). Can Microsoft really replace traditional benchmarking?

Ah, finally a place to put my 2 cents on this topic. When I saw the article I knew before reading it that Tom's will have criticised its over-simplicity. I will agree 100% that the WEI is not a replacement for benchmarking--if it was, every major benchmarking program vendor would be doing their best to sue microsoft for every penny they can get anyways.

The WEI, as the article puts it, is the WEI--IT IS NOT A HARDWARE BENCHMARK, IT IS A WINDOWS BENCHMARK. This means that DirectX versions, hardware driver versions used, HARD DRIVE FRAGMENTATION, background processes (super-duper important!), such as ANTI-VIRUS PROGRAMS, hard drive free space, available RAM, superfetched data, COLOR DEPTH/RESOLUTION, and even the amount of reboots since the last program installation WILL ALL AFFECT THE WEI FOR YOUR PC.

So...why is it different one day than the next on the same PC? Look at the reasons above. I have personally tested them all--I ran WEI then defragmented and ran again and my score changed, for a single example. Also I have some situations I plan to test:

1. Change the pagefile from one hard drive to another--does the disk index change?

2. Disable my antivirus altogether--does anything change? (tested with AVG Free 7.5 and with Norton 2007 90-day trial separately--Norton is the devil :twisted: :twisted: :twisted: , but disabling AVG completely?)

3. If I disable cool n' quiet, what will happen to my cpu's WEI?

4. Run the WEI at cold startup while speedfan confirms both cores are under 40 Celsius, then retest them at nearly 55 Celsius (by running two WoW.exe's simultaneously at max settings with two different characters--of course closing them both before testing)--how much different?

5. Disable the second cpu core in the bios, and retesting--how much performance does that second core actually give me?

Also tested my Graphics card with the newest NVIDIA driver yesterday and it is still 5.9 as it was with the previous nvidia version. Note with the ORIGINAL Vista driver my Graphics was only 5.7.

Is WEI supposed to replace traditional benchmarking? HELL no, and thank the Lord that we finally have a sytem that will be affected by all environmental conditions--so performance on one person's machine can be totally opposite someone else's experience even if the PC's hardware is 100% identical. Run the WEI and compare it to similar systems to see if you have any viruses or "resource hogs" on your system--how awesome!

Even if Vista does get plagued with viruses like XP has, now I will have an easy method to determine what the performance should be verses what it is. This WEI thing is cool.

Hmm--just thought of a new catch-phrase marketing slogan: "Got Vista, now how much does your computer WEI?" New System Requirements label:
"OS: Windows XP or Windows Vista 32-bit
For XP Systems:
Video: at least 128MB or higher DirectX 9.0c PCIe or AGP adapter including hardware T&L support
RAM: 512MB or higher recommended
CPU: Intel Pentium III 1.0GHz/AMD Athlon 1.0GHz or faster CPU
Hard drive: at least 4.4 GB free uncompressed drive space

For Vista Systems:
Video: at least 128MB or higher DirectX 9.0c graphics chipset with a WEI of 3.0 or higher
RAM: 1GB or higher RAM with a WEI of 3.0 or higher
CPU: Any cpu with a WEI of 3.5 or higher recommended
Hard Drive: Any hard drive with a WEI of 3.0 or higher and at least 4.4 GB free space"

This could be the beginning of something new and exciting, to say the least

 

[bourgeoisdude]

The numbers will go up over time, as new hardware comes out.

Thanks. Now I can sleep more easy.

More easily you mean? (LMAO sorry I just couldn't pass that up Really, just kidding)

 

[goldragon_70]

my two cents

I don't see Microsoft's benchmarking tool giving enough information, for me to decide weather it would be worth the extra money for the next level of hardware. Even looking in the article I noticed some hardware that had noticeable differences with other benchmarking software, and it barely shows up in the windows benchmarks.

 

[Zorg]

That's a laugh. :lol:

 

[MrCommunistGen]

Just my opinion, but I feel that this (like so many of M$'s other "innovations") is complete BS. An EX6800 and an 820 aren't even in the same "sport" let alone in the same ballpark... yet the WEI indicates that the EX6800 is only 14% faster.

edit1: much -> many (for grammatical purposes since this is one of my pet peeves)
edit2: Oh, and if a P4 560 scores a 5.0 I feel REALLY sorry for anyone trying to run Vista with a CPU that scores a 3.
-mcg

 

[bourgeoisdude]

Just my opinion, but I feel that this (like so many of M$'s other "innovations") is complete BS. An EX6800 and an 820 aren't even in the same "sport" let alone in the same ballpark... yet the WEI indicates that the EX6800 is only 14% faster.

edit1: much -> many (for grammatical purposes since this is one of my pet peeves)
edit2: Oh, and if a P4 560 scores a 5.0 I feel REALLY sorry for anyone trying to run Vista with a CPU that scores a 3.
-mcg

With all due respect, I believe you are missing the whole point. When it comes to performance in real-world applications and noticeable difference, the processor makes very little difference BY ITSELF. I have a Dell Dimension 4700 that now has my x700 pro video card in it, and it benchmarks the same in that pc that it did in my rig. The Dell has an old 2.8GHz Prescott D0 stepping 800MHz FSB CPU and just PC3200 DDR2 memory in dual-channel. Now-- a brand-spanking new 2.4GHz celeron would hardley score a 3.0, but a processor that is 2.4GHz or faster (2.0 GHz or faster if dual core) IMHO should score 4.0 or higher generally, except for the budget versions (celeron and Sempron).

Honestly the cpu is almost never going to be the bottleneck as slower cpus have slower chipsets meaning the memory bus is usually the culprit, so almost any Socket 775/939/940/AM2 processor will score 4.0 or higher as long as it's 2.4GHz or higher non celeron and has a chipset 800MHz or faster FSB for Intel.

So FOR WINDOWS, the processor makes little difference with a modern mobo. There just aren't many crappy cpu's that fit into the latest cpu sockets. For the record, there was a time when the cpu was fairly important even on the same platform--that was in the socket 7 era and the socket 462 (socket A) era. Today the issue revolves around the platform much more than it revolves around the cpu.

 

[surf2di4]

enewmen wrote:
Newf wrote:
The numbers will go up over time, as new hardware comes out.

Thanks. Now I can sleep more easy.


More easily you mean? (LMAO sorry I just couldn't pass that up Very Happy Really, just kidding)


Thanks. Now I can sleep more, (i.e. its) easy.

 

[niz]

WEI is nonsense. Furthermore this article is totally wrong when it says:
"Clearly, the Windows Experience Index is welcome"
because the WEI is just a thinly disguised marketing tool and gives totally misleading and blatantly incorrect scores which is worse than having no scores at all.

It doesn't actually test real performance, as I get the same score if I massively underclock my system or massively overclock it.

I have corsair PC-1111 memory (currently the fastest memory available, and then overclocked). WEI gives it a 5.5 which is the same as single-channel ddr2-667 gets. It must be reading the memory speed from default setttings or something, its certainly not actually doing any real-world performance tests.

If I didn't know better and believed the "improve your score link" I'd be screwed.
It advised that I throw away my current memory and 'upgrade' to some cheap-ass single channel ddr2-667.

I'll say it again: WEI is a piece of crap, and this is yet another Tom's article that appears to be blatantly and unjustifiably biassed towards Microsoft.

 

[bourgeoisdude]

I disagree. Again, in the real world, overclocking makes little noticeable difference for Windows performance. It makes numbers go up in "real" tests, but I have yet to hear people report any noticeable difference in everyday usage. I will probably alienate half or more of THG's readers by saying that, but when it comes down to it I've asked around and nobody plays Oblivion without overclocking then overclocks and "notices" any major difference.

I will admit really have little experience with overclocking, and with water cooled systems and such that can be overclocked to the extreme I'm sure there will be some difference. For Windows Vista, however, there is little difference in overclocking or underclocking your cpu as far as it affects performance noticeably.

Don't take my word for it, put a Pentium D 820 in your mobo and see what difference you notice in performance using the same PC. Then overclock your Pentium D. How many seconds did it save you? If the Pentium D doesn't even make that big of difference, why the heck would overclocking your X6800 make a big difference?

Again the difference is there, but a dual-core 2.4GHz or faster cpu will not make a tremendous difference in speed regardless of the flavor. Change to a different mobo, and the story might be different.

I have corsair PC-1111 memory (currently the fastest memory available, and then overclocked). WEI gives it a 5.5 which is the same as single-channel ddr2-667 gets. It must be reading the memory speed from default setttings or something, its certainly not actually doing any real-world performance tests.

It is. The memory performance going beyond ddr2-667 on that platform (honestly even ddr2-533 wouldn't be much different either) has little benefit FOR THAT PLATFORM. It's all in the motherboard and what it was designed to run at. Heck, I'll run my memory at PC1600 speed when I get home and see what happens to the score and bootup time. Both will be minimal, but I'll update with my findings anyway.

IMO, Vista is actually has the first "benchmark" that will reflect real performance. Time stuff. Prove me wrong. I'm not an arrogant in-your-face I'm smarter than you type of guy, I just want to find out if my personal experience with this is isolated just to me. Forget all the numbers, time it yourself using your watch timer.

If your PC takes even 3 seconds longer to boot after changing to slower settings from the fast ones, let me know. Then RAR 4 GB worth of data files at max settings and time it. The difference there will be more noticeable, but the average Vista user won't care about that much of a difference. Microsoft is dealing with the masses, and frankly we are part of the elite not the masses.

 

[V8VENOM]

How fortunate I am to have such a helpful tool -- is this Star Search or American Idol -- or maybe it's Survivor? If Microslop score you low enough the OS votes you out it's validation database and you can't re-install til you upgrade to atleast a 12Ghz 8-core CPU, 32GB of RAM, and 4 GPUs -- I mean it was built for future hardware right?! Because it does so much more for you, like...like...errrr...like...

So Vista is going to be the dominant OS by mid this year?? Well there are some real world market research polls that would make that statement False. THG has gotta stop humping Microsoft's leg -- don't worry THG, you'll get your turn at being bought out as soon as you become significant enough.

Other than my many installs and uninstalls of Vista for testing my software, I don't know of anyone that is planning on buying Vista within at least the next 12-18 months (if at all).

No thanks Vista, I'll use 3rd party tools or code my own to get real performance numbers -- go ahead, vote me off the Island.

Rob.

 

[gr8r-x]

Can someone explain what those MS numbers mean?
Ok, 1 is lowest and 5 is hghest. You need X benchmark to play Y game requiring an X.
But what happends in a few years? The Playing tomorrows software on todays hardware will not work well.
Do you expect numbers will keep going up, 6,7,8,9, etc?
That way a '4' in 2007 will be the same as a '4' in 2010.

I underatand the Windows Experience Index us just a rough guess so the casual buyer won't need to read so much gobbledygook about "System Requirements:"

They're scored between 0 and 6. 8)

And I'd have to agree that it is a welcome addition, certainly for a quick "how crap is this sytem really" test. Think about all those times you get called to relatives/friends/customers houses and they say that their PC is really slow. Now you can start up WEI, have it report back some really bad numbers and just point at the screen and say "See, Windows says that your hardware isn't good enough".

Soooo much easier than arguing that yes, they did buy the PC *only* a year ago, but it was 2 years old when they bought it.

And the whinging about how a Pentium D scored close to a C2D, who cares? I've just "upgraded" to an AMD 64 3400+ and it's, personally, fast enough for me and my games. Sure, all these dual/quad-cores are lightning fast, but unless you are doing some seriously time-critical work, is it REALLY that important?

Windows boots 5 seconds quicker. Word opens 10 seconds faster. WoW gives me 3 more FPS. And my bank balance is a thousand dollars lighter..... I know what I value more. :lol:


PS. I've installed Vista 2 times. I've also reformatted and installed XP 2 times.. Very unimpressed. To quote comic-book guy: Worst OS .. ever!

 

[randomizer]

What a pointless article. Its obvious that M$ pathetic software isnt going to replace "traditional" ("real" I prefer) benchmarking. How can a number between 0 and 6 tell you anywhere near as much as real framerates and real decoding times etc?

 

[db101]

For Vista Systems:
Video: at least 128MB or higher DirectX 9.0c graphics chipset with a WEI of 3.0 or higher
RAM: 1GB or higher RAM with a WEI of 3.0 or higher
CPU: Any cpu with a WEI of 3.5 or higher recommended
Hard Drive: Any hard drive with a WEI of 3.0 or higher and at least 4.4 GB free space"

This could be the beginning of something new and exciting, to say the least

This is a good example of what some program/game could have on the box... we've all seen this coming. The hardware is gonna need a WEI score as well then. Because, say someone who is not computer-savvy goes to get some cool new game and sees that it requires a graphics score of 4.0 or higher. So he(/she) heads over to the graphics card isle in Best Buy ("Worst Buy") to see what kind of product is needed for said game. The WEI score on the game box is useless in picking out a new graphics card if the GC box doesn't post a WEI score. But a score can change, depending on the mobo chipset and the like, right? I sense a disturbance in the force about this...

 

[enewmen]

The numbers will go up over time, as new hardware comes out.

Thanks. Now I can sleep more easy.

More easily you mean? (LMAO sorry I just couldn't pass that up Really, just kidding)
I corrected the post and I regret the error.
Now I can finally sleep more EASILY.

 

[randomizer]

Cant let Tom Baker see typos like that. Expect to see the TARDIS in your backyard.

 

[Newf]