Microsoft Tussles With Consumer Reports Over Surface Reliability

Should you buy a Surface laptop or tablet? Well, that's going to depend on who you ask. Consumer Reports doesn't think so. The magazine pulled recommendations for four Surface products—two laptops and two tablets—because it believed 25% of the devices would present their owners with some kind of problem within just two years. Microsoft responded with a statement that said its data refutes these findings.

Determining which side is right would be like peering into a dark room and trying to identify two different shades of black paint—it just isn't happening. Consumer Reports didn't reveal many details about the reasoning behind its decision; the most it said was that Microsoft's hardware was more likely to break than products from other brands, according to estimates it made after surveying its readers about their personal experiences. Because Microsoft's defense is based on internal data, we can't verify that Surface products fare better in the wild than on paper.

Consumer Reports' Vague Estimates

Let's start with Consumer Reports. The magazine noted that its decision wasn't the result of testing: "Several Microsoft products have performed well in CR labs, including the new Microsoft Surface Pro, which earned Very Good or Excellent scores in multiple CR tests," the outlet said in its post. "Based purely on lab performance, the Surface Pro is highly rated when used either as a tablet or with a keyboard attached." The problem, it said, was that shoppers care just as much about long-term reliability as they do about how a device works out of the box.

That's where things get tricky. Consumer Reports said it based its estimates for breakage rates on a reader survey, but it didn't say how many people reported problems, how that number compares to the total number of Surface product owners, or how much more likely Microsoft's hardware is to suffer issues than competitive products. Instead, it said there are "millions" of people who respond to its surveys to share their experience with "hundreds of thousands of products" in various categories. Neither figure is particularly helpful in this specific case.

Nor is the crux of Consumer Reports' advisory:

A number of survey respondents said they experienced problems with their devices during startup. A few commented that their machines froze or shut down unexpectedly, and several others told CR that the touch screens weren’t responsive enough. [...] The new studies of laptop and tablet reliability leverage data on 90,741 tablets and laptops that subscribers bought new between 2014 and the beginning of 2017. Predicted reliability is a projection of how new models from each brand will fare, based on data from models already in users’ hands.

"A number" and "a few" aren't hard numbers. And without knowing how Consumer Reports made its estimates, how it determined the data gleaned from the reader survey is accurate, or how Surface products compare to others, the figures the magazine did share aren't that useful. Consumer Reports could be right, and it was confident enough to raise Microsoft's ire, but too little information was shared for outside observers to understand its thinking. Whether or not this should stop you from buying a Surface product depends on how much you trust Consumer Reports.

Microsoft's Slightly Clearer Counterpoint

Microsoft didn't offer much more information in its response. Here's the pertinent bit from its blog post about standing behind Surface:

In the Surface team we track quality constantly, using metrics that include failure and return rates – both our predicted 1-2-year failure and actual return rates for Surface Pro 4 and Surface Book are significantly lower than 25%. Additionally, we track other indicators of quality such as incidents per unit (IPU), which have improved from generation to generation and are now at record lows of well below 1%. [...] Surface also ranks highly in customer satisfaction. 98% of Surface Pro 4 users and Surface Book users say they are satisfied with their device*, and our Surface Laptop and new Surface Pro continue to get rave reviews.

That offers some more hard numbers, but it still doesn't tell the whole story, and we don't know how accurate the company's internal data is. If the failure and return rates are "significantly lower than 25%," and the number of incidents per unit is "well below 1%," why not give the actual figures? Speaking in percentages also helps downplay the issue—saying that fewer than 25% of the people in a room saw you pick your nose could mean that 100 people saw you go nostril-diving. That's not a lot percentage-wise, but it's probably more than you'd like, isn't it?

This back-and-forth highlights the problems many people face when it comes time to spend their hard-earned money on a laptop or tablet. Reviews can help (that's why we publish them), but it can be hard to predict how well a device will hold up over the years. Consumer Reports made an effort to balance what it sees in the lab, which says that Surface products are good to go, and what it predicts will happen down the line, which is less favorable. Microsoft claims those estimates are incorrect and appears to have the numbers to back that claim up.