Scope and relevance of testing

Tom’s Hardware is an important part of my knowledge base with regards to the PC industry so nothing I’m going to say here really changes that. Your recent test of the ATI 38xx series Video cards against the Nvidia 88xx/8600 series, particularly the 8800 GTX card was basically irrelevant to the bulk of the market place and I suspect most of Tom’s Hardware readers. The tester did a good job so don’t misread that but the focus of the testing is so far out of scope and relevance for the bulk of the PC market consumers as to be almost entirely moot. Consider the following.

If you change the players to cars instead of Video game cards and insert these four models, Porsche 911 Turbo, Nissan V6 Maxima, Honda Accord V6 Coupe and Toyota Camary V6 Coup the bulk of the car buyers would recognize the pointlessness of including the Porsche in the test. How many Maxima’s, Accord Coups or Toyota Camarys can you buy for one Porsche 911 Turbo? Same for the Nivdia 8800 GTX vs all the other cards? The title of the article should have been “The Honda Accord V6 strikes back against the Porsche 911 Turbo” just to be accurate. Value can not be separated from performance for 99% of the market place.

The second problem covers scope. My friend’s 27 year old adult son complains about the performance of his 1 ghz Athlon all the time. The system is as fast today as the day he bought it so the real reason he complains is because he can’t get any of the newer $50.00 games to play on it. When his 64 MB graphic card died (an Nvidia) I gave him a spare 4x AGP 128 MB ATI Radeon 8500 I had. Still complains. His parents won’t buy him a new $3000.00 game system to play the $50.00 games he wants. Clearly outside the scope of Tom’s Hardware to solve but it is clearly obvious that most of the games you review at the resolutions/color depths you do with the features you turn on are outside the mainstream (aka ASP) performance levels of systems today. My 2003 systems have the second highest 8x AGP rated video card (in 2003 terms) and would be hard pressed to handle Vista and any of the games you review but the systems are just as fast today as then (faster actually) and do everything I want them to do as fast as is practical. The two WD Raptors I added to each in RAID 0 speeded up the system across the board and while it is true the drives are over priced, as one of your reviewers also stated, the benefit covers a lot of ground. I couldn’t tell the different between the Nivdia 8800 GTX and the 256 MB ATI card I have in my desktops in daily use nor would I or could I run the monitors to the resolution levels you do. Stated another way, you need about a $3000.00 system to get the best out of an 8800 GTX based system. This is born out by your latest Cyrsis testing which essentially required a QUAD Core CPU system and at least one 8800 GTX to get minimum performance levels. SLI with multiple GTXs would be required to get any head room which won’t work with the new 45 MM Intel Quads. More money…..

At the end of the day, those that have already made the investment in a 8800 GTX based system already know the results of the testing; those with the lesser cards also know the results and all that leaves is the rest of the market where funds aren’t unlimited and buying a complete new system to get acceptable performance from one of the reviewed games isn’t high on the option list. You could replace the Video card test with the same for CPU or Solid State drive tests also. The scope and relevance of the testing covers a very small part of the market place and if I were a game producer I might be concerned about the conclusion one might draw about the over all test results. Why would anyone buy my game if it required thousands of dollars worth of upgrades to get acceptable performance? What is acceptable performance? I run at 1024 x 768, 1280 x 960 with 16 bit color. There is no point going beyond 16 bit color and the extreme resolutions you run at are limited to very high end monitors and stress them a bit.

As I said, the tester did a good job but this is what I conclude from such tests (includes most CPU testing also). The games tested are outside my scope of use or desire. The games are either poorly coded or simply targeted for a tiny segment of the market. They could give the game away and I couldn’t afford to play it given the test results. Same for the lastest CPU tests. If I need to do those specific things outlined in the CPU test day in and day out and money to replace a system every 6 months or so isn’t limited then the test results have merit. I can’t tell the difference between my 1.6 Pent M laptop and my Athlon 3200s on the Net. The benchmarks I run say there is a huge difference. Some things I do demand the desktop do it vs the laptop. I could more than double my CPU performance tomorrow by replacing my system board, memory, CPUs and video card. That would cost over a $1000.00 per machine without new monitors that could run this high resolution. That would bring this up to $1500.00 per machine just to get acceptable game performance from Cyrsis? Those with 8800 GTXs will be relieved that they don’t have upgrade to play Cyrsis.

Value and scope are important to potential buyers of PC equipment. Extreme testing like is done with Video cards and CPUs tend to focus only on the top 1% of the market place. Nothing I would buy off the shelf at Costco, SAM’s etc would pass your Video card tests and for most buyers that has no relevance in the near or distant future. I understand the games tested are popular with “Gamers” which is limited to 3D games and of particular FPS type action games. I also understand that it took 256 graphic PCs running a couple months to do Titanic’s special effects. Sinking what is required to play some of these games today would result in a Titanic effect on our budgets and cause us to disappear below the waves for a long time if we made such a decision simply for a game. Perhaps we could get extreme equipment testing moved to the Titanic section where such fleeting benefits could be viewed by those that have unlimited funds. Other than that, a little more scope and relevance to the average PC purchaser would be greatly appreciated.