Why did the quality of Tom's Hardware GPU reviews went somewhat downhill?

[refillable]

Hi fellow THW citizens.

This is kind of like a personal rant on how the situation has been for the past one or two years. Before you fly personal insults on me, I'd like to introduce myself first. I've been a Tom's Hardware user for almost five years now, got some badges and had been quite active for a while. Now, even though not as active on the forums, I'm still coming here for reviews and those neat historical slideshows and well written guides. You can check my profile if you don't believe me, I hope I'm not going to damage my reputation just by posting a controllable rant.

It begins with this 290X review, 3 years ago. In my perspective, this is where I started think Tom's Hardware needs a new overhaul of the reviewing system (I did however found this as well somewhere in the GTX 680/7970 ages, but nevertheless nothing major). At that time, the 290X was obviously dominating and it was very easy to draw out the obvious conclusion. However, after this 780 Ti review, it was well noted that the 290X that was once known to be great, struggled against other Nvidia solutions that it once beat, because of "adjusted clocks using the reference cooler".

This made me chuckle a bit, because I know for a fact that it's going to make a lot of people confused, because you put all the focus into the reference version, while only mentioning a glimpse about AIB partner coolers, that're simply going to make a night and day difference. This, at that time, was I think extremely critical, as back then the market didn't know the options as well as it does nowadays.

Moving a generation further, we all saw this 980/970 review coming out, I then started to question your game choice, where you included 7 games and two of them, Assassin Creed and Watch Dogs, embarrassingly unoptimized. This means that from your only 7 games, only 5 are qualified enough to draw out a comprehensive conclusion, which I think isn't even enough to make a tight winner between the cards - let alone changing people's buying decision.

Things seemed to get worse when this 390X/380/370 review came out. You didn't even praise a single thing that is good about the cards! I know they're re-brands, I know that AMD hadn't been innovating that time, but hey, what about the key part of you guys reviewing? Where's the price to performance ratio there? Yet, it said

"AMD’s Radeon R7 370 is actually the most palatable of its three models, competing readily in the entry-level segment using acceptable efficiency, in spite of being three years old."

Which doesn't take a lot of thought process to conclude that there had been something wrong! Because the R9 390 and the R9 380 at a similar price point trounced the GTX 970 and the GTX 960 respectively!

Continuing to the R9 Fury X vs the GTX 980 Ti things might have improved, no broken games and everything was pretty much going very well. But still, seven games are still to little, and I had no idea why you guys removed the workstation/GPGPU benchmarks, but I wished these reviews continued with the addition of the number of games tested.

But it didn't happen.

This GTX 1060 review is where it all went downhill,

Broken games are back! (Project CARS, yay!)
BF4 (almost 3 years old!)

And what baffled me the most was:

"Averaging out the percentage differences between them, GeForce GTX 1060 is about 13.5% quicker at 1920x1080 across our suite. Its advantage slips to 12.5% at 2560x1440"

An average that included a 3 year old game and a super broken one, now that's what I call misinformation. And mind you that this is only 9 games, where are the other ones that at least isn't from yesteryear? There're thousands to choose for AAA games, why do you have to choose BF4 or Project CARS? I shook my head. Also, you must have been quite informed about future Low Level APIs, where are the games that has that? You used three out of nine, that only changed things a little, while the conclusion readers can take if there were more of them is much more significant than that?

And to sum that up, this is the one that made me face palm.

http://www.tomshardware.com/reviews/gpu-hierarchy,4388.html

In all, I'm not flaming Tom's Hardware, I'm not hating them, I'm not attacking them personally. But this is just how I see it. I know you guys have a huge audience, a huge reader base and a popular site overall. Just think about how the mistakes could change buying decisions of thousands! I know I've been keen to speak up about this, but I'd choose to say it at the right time, now. This is getting pretty intolerable that I've to speak up. I'll say sorry if I offended you in this essay, but please hear me and, feel free to disagree, in the correct way.

Thanks.

 

[boju]

Just a suggestion, might want to post your view on the matter in the forum feedback instead of here in the graphics's card forum. Will just get lost here.

 

[Plumboby]

yeah more on the lower less knowing 3rd party video card brands as end or the day to inform us what else is on the market to try to offer

 

[refillable]

I admit I was completely wrong on the section, but how do you move it?

 

[refillable]

Well it seems that they don't bother. I'm going to see what's gonna happen in the next review.

 

[bigpinkdragon286]

One change that might not be too drastic but would certainly help with the graphics card reviews, instead of giving us an average percentage of games chosen by the Tom's staff, let the reader choose.

Tom's has no concern about slapping tons of slick graphic slides into their articles, so I doubt it would hurt to add one more with check boxes for each game benched and a total average. Don't forget to make resolution a choice as well. This would mitigate the problem of continuing to include games like Project Cars. Readers could uncheck irrelevant titles and get a more realistic view of what they plan to do with the cards.

 

[FritzEiv]

@refillable (and everyone else here): I see your comments. So first of all, thank you for taking the time to make your voices heard. I'm in the middle of a series of pre-holiday planning, budgeting, and the normal things that companies go through at this time of year, so I'm going to keep this first reply short, and I'm going to talk with a couple of our testers about all of this, and then in short order I hope we'll be able to provide a better response.

Right now we're working on publishing roundups of third party cards (1080s, 1070s, and 1060s). We've actually published these in Germany, already and we've translated them and are now editing them and piecing them together for final production. You should see the 1080 roundup in the next couple of weeks. The games we tested are: Ashes of the Singularity, Battlefield 4, Project CARS, The Division, Grand Theft Auto V, Hitman (2016), Rise of the Tomb Raider, The Witcher 3. I hear you on Project Cars and we have been talking to some folks about that one.

Sometimes we keep games for consistency and comparability. Stating the obvious: changing games constantly wouldn't give us consistent results. However, we have -- based on some of our analysis and feedback -- created a new test suite that we are just starting to roll out, and which you'll see shortly when we launch our gaming laptop reviews. We have painstakingly sought out games with built in benchmarks, and we have ensured that there are an equal number of games representing Nvidia and AMD "sponsored" titles, so to speak.

One last thing for now: I sense that the AMD folk among you would probably ask "when are the AMD roundups going to occur" and it would also be fair to ask about that on the gaming laptop side. The truth is that we haven't had much third party support here. Before reading this thread, I'd made a note to go to AMD and ask if they can get their partners moving on getting us cards (and laptops) to test. I'll report back when I have a satisfactory answer.

More to come, and please don't think we're ignoring you. We're being deliberate about changes, but we do discuss what we hear, and we do respond, and we are committed to improving what we do and satisfying all of our readers to the degree that it's practical (setting aside the request to "test this card at X settings and Y resolution with Z configuration on <insert favorite game here>" -- and yes, we get that quite a bit, minus my exaggeration).

 

[refillable]

Good to hear that you're not ignoring us. Please do remember that we're just doing some criticism, not a fanboy rage or any sort. It seems that you're fully aware of this.

Let me list my objections (in case you guys missed some of them)
-Be wise on the game choice, inconsistent games (I meant, the way that the results differ from other games with respect to the different GPUs) such as CARS can swing the average a lot, such as in the GTX 1060 review. I reckon the 1060 is only going to be around 8-9% faster rather than 13.5% when CARS is replaced with another game.
-If you really want to keep games like CARS for its popularity, the consequence is to add 5 to 10 equally demanding games.
-Stop focusing on ONLY the reference cards, mention that AIB partner's cards should fix the problem without any significant side effect to gamers and enthusiasts alike.

Now as far as how consistent or comparable the games are, there's no way to achieve this with 7 nor 9 games. You should get AMD AIB partners to cooperate, but it's nice to hear that you're trying to make that happen.

I'm hoping that you are going to change a great deal of things positively in the Vega/Pascal refresh reviews.