Vista: Benchmarking or Benchmarketing?

Memory

We ran the memory benchmark of SiSoft Sandra 2007 and the CPU benchmark of PCMark05 to verify the impact of memory size, speed and configuration. Here are the setups we tested:

DDR2-667, 512 MB Single Channel
DDR2-667, 1 GB Single Channel
DDR2-667, 1 GB Dual Channel
DDR2-667, 2 GB Dual Channel

DDR2-800, 512 MB Single Channel
DDR2-800, 1 GB Single Channel
DDR2-800, 1 GB Dual Channel
DDR2-800, 2 GB Dual Channel

In this case, the Windows Experience Index shows clearly more differences than our benchmarks, as the WEI takes into account the fact that Windows Vista runs noticeably more slowly with 512 MB RAM. Apart from that, the memory setups at 1 GB and 2 GB RAM reach similar performance indices. Single or dual channel setups make a difference in memory benchmarks, but not for Vista's memory index.

Graphics

We used Call of Duty 2 and 3DMark06 to compare to the Experience Index. 3DMark06 is a popular, though not undisputed synthetic benchmarking tool, but it also can be seen as a de-facto industry standard. Call of Duty 2 is a 3D shooter, which can represent average 3D games today.

We selected five different graphics cards to represent high-end, mainstream and entry-level graphics capabilities. The Nvidia GeForce 8800 GTS represents the high-end segment, followed close behind by ATI's Radeon X1900XTX. The GeForce 8 card scores an index of 5.9 both in the Windows graphics and gaming graphics categories, while the Radeon X1900XTX reaches 5.9 for Windows and 5.8 for gaming. Since the card certainly offers sufficient performance for anything you want to play, the reason for missing the 5.9 top score is the lack of DirectX10 support, which Nvidia's GeForce 8 series has.

The next card on the list is an ATI Radeon X850XT, which is one generation behind. It offers high performance, but doesn't support Shader Model 3 (DirectX 9.0c). For this reason, it misses the level 5 score and only rates at an experience index 4.9. For Windows, this card still hits the maximum of 5.9. The situation is similar for the Nvidia GeForce 6800GT, which is an early DirectX 9 card and doesn't perform as well as today's cards. It scores 4.5 for gaming and 5.9 for Windows graphics. Finally, we installed an older ATI Radeon X700 Pro, which also scores 4.5 index points for gaming and 4.9 points for Windows. It isn't DirectX 9 compliant, which is why it doesn't support transparency features for Windows Vista's AeroGlass interface.

Again, we were pretty sure that the game and 3DMark benchmark results show more substantial differences, and we were right. While the gaming Experience Index of all five cards never drops below 4.5, which indicates at least good performance, our benchmark results are dramatically different. At a resolution of 1280x1024, which is the most popular resolution for TFT displays between 17" and 19" today, Call of Duty 2 is almost unplayable with the GeForce 6800 GT and ATI's Radeon X700 Pro, and even the Radeon X850XT only reaches 32.4 frames per second maximum. The two top-of-the-line cards - the Radeon X1900XTX and GeForce 8800GTS - reach 66 fps and 92.9 fps, which certainly is more than enough for a smooth gaming experience.