Valve Fixes The Steam Hardware Survey

Valve identified a problem with its Steam Hardware Survey and has deployed a fix. The survey tracks a portion of Steam's 125 million active users, so we in the media often uses it to track the hardware and software that gamers use most frequently. Unfortunately, the widely cited survey has suffered from extreme changes in key tracking areas over the last seven months due to an error in the reporting system. That means the survey has largely been useless for its intended purpose of reflecting broad trends in CPU and GPU usage, among other areas.

Steam's survey asks users to agree to participate, and if a user opts in, the survey queries the system for basic information about the hardware and software installed on the system. Valve adds the information to its database and then shares some of the most important data with the public.

Steam designed the survey to query its users’ systems once per year, but the company discovered that Asia-based cyber cafes manage their systems in such a way that the survey could be completed multiple times per year on a single PC. This duplicated the entries multiple times, which skewed the results of the survey. Typically, we would expect an automated system to weed out multiple entries through identifying information from the host computer, but Steam either protects its users by not collecting such data, or the duplicate entries simply went undetected.

The duplicate entries led to inflated statistics for Windows 7 usage, CPU and GPU market share, and an erroneous report of a rise in the number of quad-core systems. Steam has deployed a fix to correct the issue, and the company contends that all its data from April 2018 onward will be correct.

However, it appears that the company isn't correcting the erratic data it gathered in the previous months. That means the April 2018 Steam Hardware and Software Survey is the only accurate measure of hardware usage by Steam members in the last seven months. It also isn't clear if the company made other changes to its survey methodology since it last posted accurate data in July 2017. 

AMD's return to prominence in the desktop processor market finds it trading blows with Intel as they release new waves of competing chips, and while it's difficult to find accurate market share information, the company has undoubtedly made up some ground. The latest survey doesn’t reflect that change. Assuming Steam's data in July 2017 was accurate, this chart outlines the changes over the previous seven months:

Swipe to scroll horizontally
Steam User Market ShareAMD CPUIntel CPUAMD GPUNvidia GPUIntel GPU
Steam July 2017 Survey19.42%80.49%20.23%53.9%15.52%
Steam April 2018 Survey15.96%84.04%14.89%75.26%9.69%
Increase/Decrease-3.46%+3.55%-5.34%+11.36%-5.83%

Given the success of Ryzen, the 3.46% reduction in systems with AMD processors seems questionable. Steam doesn't share much public information about its survey methodologies, such as how many systems it queries per year or the percentage of laptops and desktop PCs, so we can’t use these numbers as a direct comparison of AMD's penetration into gaming rigs.

The GPU market is also tricky to accurately measure, especially given the enormous number of GPUs that have found a home in mining rigs. In either case, the survey results claim that AMD has suffered a 5.34% decline in Steam users' gaming machines, while Nvidia has made significant headway against AMD and Intel (iGPU).

Steam's database also tracks other key metrics, such as how many CPU cores the average system has, Windows distributions, gaming resolutions, and memory capacity, among others. While these measurements are also subject to many of the same caveats as the rest of the Steam survey, they're the best available measure of trends in the gaming market.

Unfortunately, for now, we're limited to the April 2018 results alone. Because the statistics for preceding months are skewed wildly, we won't know the true monthly changes until next month when Valve releases more statistics with the fixed reporting system. That means you should ignore many of the recent articles that analyze the month-over-month changes.

Paul Alcorn
Managing Editor: News and Emerging Tech

Paul Alcorn is the Managing Editor: News and Emerging Tech for Tom's Hardware US. He also writes news and reviews on CPUs, storage, and enterprise hardware.

  • bskchaos
    Unless steam track if you actually play games on the systems the survey is kind of useless. I have 4 machines with Intel CPUs but only one is used to play.
    Reply
  • SkyBill40
    I don't keep Steam active all the time and though I do have some games on it, I can't remember the last time I played one of them. So, it's essentially missing my data.
    Reply
  • derekullo
    I'm assuming its a program similar to Deepfreeze or Centurion that was causing it.

    Why is this specific to Asia?
    Reply
  • 10tacle
    20942388 said:
    I don't keep Steam active all the time and though I do have some games on it, I can't remember the last time I played one of them. So, it's essentially missing my data.

    Well if you don't regularly game on Steam then the point of your input is rather useless for statistics. It's like GM taking a survey on who has both a Chevy and Ford in his garage. If the Chevy is parked more than the Ford, does that mean the Chevy is not used? If one is not an active Steam gamer, then said individual's input, with all due respect, is irrelevant. People in the marketing business know this well.
    Reply
  • cryoburner
    20942274 said:
    Assuming Steam's data in July 2017 was accurate, this chart outlines the changes over the previous seven months:

    Given the success of Ryzen, the 3.46% reduction in systems with AMD processors seems questionable. Steam doesn't share much public information about its survey methodologies, such as how many systems it queries per year or the percentage of laptops and desktop PCs, so we can’t use these numbers as a direct comparison of AMD's penetration into gaming rigs.
    While they might have potentially addressed the issue of systems in Chinese net cafes getting counted more times than intended, the overall demographic of users running Steam has still changed. There is still a notable increase in the number of users running Steam in that country compared to before.

    According to the current survey, 30.35% of systems participating in the survey have their language set to Simplified Chinese. That's half the percentage reported a couple months back, but still nearly double what it was in the July 2017 survey, when it was at 16.64%. Likewise, the percentage of systems running Windows 7 64 bit is still getting reported as being higher than it was in July of last year. The results might not be getting thrown off quite as much by net cafes, but they are still affected by a change in demographic.

    This should highlight the fact that these Steam surveys should not be taken as a precise measurement of hardware trends, since the pool of systems taking part in the survey can change from one month to the next. In this case, Steam's growth expanded in China, and since net cafes are a popular way to play games there, the results came to reflect the hardware being run in Chinese net cafes more than anything. These kinds of variances will continue to occur as Steam gains popularity in various countries, and since hardware can vary by country, that can potentially skew the numbers. Many of the results can still be considered useful for various purposes, but there are too many variables getting shifted around in the background to treat them as anything more than rough estimates covering an ever-changing international user-base. Perhaps if Steam provided additional filtering options, such as limiting results to various countries, the results might more accurately depict trends within those regions, but they have not provided that option as of yet.
    Reply
  • alextheblue
    20942296 said:
    Unless steam track if you actually play games on the systems the survey is kind of useless. I have 4 machines with Intel CPUs but only one is used to play.
    Well they're not going to be surveyed if they don't have Steam installed. But if you DO have it installed but you don't game much on that device, it's hard to correct for. For example, I have Steam installed on a couple of machines but only one of them runs any demanding games. The other one (older laptop) only runs the occasional lightweight game. I certainly wouldn't submit the specs of the laptop, it's a clunker. But where do you draw the line? What if someone plays some casual games here and there on such a low-spec device? Are their results even relevant? I'd personally say no, not really... their result might as well say "dual core toaster oven" but it fits your definition as a valid survey target.

    The bottom line is anyone who views Steam survey results as a measure of gaming PCs need to reconsider. It's a lot less useful dataset than people have been lead to believe.
    Reply
  • Brian_R170
    The decrease in AMD CPUs may be accurate, but skewed by the shear number of "old" systems running steam.

    Example: say all old systems get scrapped when they are exactly 10 years old. If AMD's market share was higher 10 years ago than it is today, then even though AMD's market share is may be increasing today, the total number of systems with AMD CPUs running steam will still go down.

    Example: say it's much more likely that old systems with AMD CPUs either get scrapped or re-purposed for non-gaming compared to old Intel systems which continue to be use for steam.

    Still, it's just hypothetical. I'd be more inclined to believe it's due to there being a much higher number of laptops sold compared to desktops and a much higher number of laptops having Intel CPUs compared to AMD CPUs.
    Reply
  • 10tacle
    Again, we are not talking about a professional scientific managed poll here. It's not a controlled polling environment. It's on a volunteer only basis. Regarding older AMD processors and whatnot, nobody is playing current Steam AAA title games on them so from my viewpoint, that is also worthless data. What I want to know is a snapshot of user hardware who are active on Steam gameplay (or GOG, or the other game portals for that matter).

    We will never have perfect data captures because as mentioned above, you have people who are Steam account users who have old hardware that can't handle modern titles. Then you have Steam users who are retro gamers and play 10+ year old titles on top of that. From my seat, the only valuable information would be to poll the games the users are playing and their hardware. You can't play Far Cry 5 on an Intel Core 2 Duo E8300 running a GTX 275 at 1080p. Or at the very least, separate modern vs. old hardware and see what resolutions and games they are playing at.

    For now, what Steam has is the best snapshot overall. Can it be improved upon? Of course it can. Oh and speaking of 1080p resolution, it's still the plurality of monitor resolutions that gamers are playing with. By far.
    Reply
  • rwinches
    Participation is optional so we need to know what percentage of total steam users are part of the survey. They could just give an option to enter CPU/GPU and OS info to make that portion more useful.
    I can see why users hesitate to allow access, it's bad enough that MS tracks things like how many apps/programs installed added or removed etc.
    Reply
  • 2Be_or_Not2Be
    The survey also has hardware identification issues. For example, I have a mini-ITX system that was classified as a laptop. Since you can't correct it manually, I wonder how many other data points were from incorrectly identified components.
    Reply