Some Feedback on Toms Hardware CPU reviewing

Status
Not open for further replies.

Intrepid_1

Prominent
May 5, 2017
24
0
510
Let me just start by saying i want this feedback to be constructive, please no hate or bashing :)

I will try to keep it as short as possible, tho there is a lot to get through.

There are a couple of things to know about CPU performance testing for Games, the most fundamental and basic of which is to insure the performance you are getting from the system is bound by the CPU and not the GPU.

If the GPU is doing most or all of the work then you are not seeing how much performance the CPU has to give, make sense, right? :)

To me Toms Hardware's Ryzen reviews fail to do that.

For their game testing they are using a GTX 1080 GPU, 2667Mhz System RAM and the highest game quality settings possible. Ultra High quality settings @ 1080P in BF1, for example..... now i have a GTX 1070 OC to run at around stock GTX 1080 performance and i know in campaign mode at those setting the CPU is not being stressed at all.

As a result a CPU with higher single threaded performance, such as a higher clocked CPU like the 7700K will win, predictably that is the conclusion Toms arrived at, nothing at all wrong with that result.

Its the methodology are reasoning behind it that's flawed.

Most of us keep our CPU's for 3 to 5 years, the thing that we upgrade is the GPU, we do that because they get faster, games get more advanced and use more system recourse as a whole, so what i want to know is how much longevity is in the CPU, how much headroom is left in it?

When you look at reviews that do actually stress the CPU what you see is that often (not always) but significantly is that the equivalent Intel CPU's are nothing like as powerful as the Ryzen CPU's in a lot of games.

I think the problem is an assumption that games do not use more than 3 or 4 threads, that is an out of date idea, its an idea that should have been put to rest when you realised that back in the day a 4 core 2500K was always 20% or so faster than the FX-8350, now its often the other way round.

I can easily illustrate this with what is a well executed CPU review.

We all know Metro Last Light, how CPU Intensive it can be, well look at this.....

That is a massive 55% performance advantage to the 6 core Ryzen over the 4 core Intel, a ridiculous performance win for AMD,
why? well this is why a video run-through review is so much more telling than numbers on a slide, look at the thread load on the Intel CPU on the right vs the AMD CPU on the left, all of the Intel's 4 threads are completely saturated, as a result the GPU is over 50% more bottlenecked on it than it is on the Ryzen CPU.

4NQORdh.png


Now, to be even more clear about this,

First image both CPU's at 249 FPS, i caught it just where the Intel CPU was at its limit.

Xz6lcKg.png


Moving on a little the Intel CPU still at 100% but the game wants more, to the Ryzen chip now pulls ahead, 318 FPS vs 277.

ZiKe7di.png


Moving yet more and the Ryzen CPU is still gathering even more pace....

eXunnst.png


The truth is when actually put to the test AMD's Ryzen CPU is not just faster than Intel's rival, it destroys it....

Here is the Ryzen 1700 @ 3.9Ghz vs the 7700K @ 5Ghz in BF1, yes what you are seeing there is the 7700K's 8 threads completely maxed out.

CrM80t4.png



I would like to see Toms Hardware to change its testing methodology to reflect modern day realities, so to give its readers accurate advice.

Thank you :)

https://www.youtube.com/watch?v=WlOs_McAVZ0

https://www.youtube.com/watch?v=BXVIPo_qbc4
 
You know where is a big flaw in your approach? No one buys 200$+ CPU and 400$+ GPU to play at 1280x720 low settings ... When you spend so much on comp, you expect to play at highest possible settings and resolutions, that's why the tests are run at such specs as they are.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510


That is an argument the AMD Bulldozer crowed used to make when reviewers used exactly the same testing methodology i'm advocating, the reasoning being as i explained: by not loading the work onto the CPU you are not measuring its performance, there in it is not an accurate representation of the CPU's performance, the CPU's actual performance matters because games use more of it as time goes on and GPU's become faster.

Perhaps a compromise would be to do it both ways?

 

Intrepid_1

Prominent
May 5, 2017
24
0
510


Well actually what you can do is save your self $150 by opting for the 6 core Ryzen instead and spending what you saved on a more powerful GPU.

A GTX 1070 instead of a GTX 1060, perhaps...

Thats good consumer advice.
 

4745454b

Titan
Moderator
I'll take a stab at this, why not.

by not loading the work onto the CPU you are not measuring its performance, there in it is not an accurate representation of the CPU's performance, the CPU's actual performance matters because games use more of it as time goes on and GPU's become faster.

Or in other words by increasing the resolution you are stressing the GPU and not the CPU. So in a CPU test you should put the burden on the CPU. Not bad in theory. The issue with this of course is you can get odd results compared to what you get when testing normal resolutions you would actually play at, 1080, 4k, etc. Case in point is your 720 results which show the 6600 gets "destroys". Probably is what happens when you play a game on your 1080 or 4k monitor?

aHR0cDovL21lZGlhLmJlc3RvZm1pY3JvLmNvbS8zLzcvNjY3NTU1L29yaWdpbmFsL2ltYWdlMDQ4LnBuZw==


Oops. Now the 1600X even OC'd isn't as fast as the i5 7500. I get that you need put the stress on the part you are testing. But "real world" use matters as well. Synthetic testing and be used to show how the memory controllers are different, or how different cache hits are, but who cares if the memory controller is 15% faster if real world usage shows no difference in zipping files, or loading programs? 1080 and 4k gaming results is what info we need.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510


Edit: actually you hit the nail on the head, yes it matters as well, but the CPU true performance also matters....
The chart you posted is exactly my point, the Intel CPU is good or better for as long as the GPU performance is not higher, if you increase the GPU performance the low threaded performance becomes irrelevant as higher GPU performance adds more and more work onto the CPU, when that happens the Intel CPU only having one third the amount of threads becomes saturated, it can offer up no more performance to keep up with the GPU power, eventually as you up the GPU performance the Ryzen chip with more compute threads overtakes the lower threaded Intel chip, it keep gathering pace while the Intel CPU remains saturated with nothing more to give.
This to such an extent that 'as we see in the screen caps' the 6 core Ryzen is as much as ~60% faster than the 4 core Intel i5.

I get that you want to review real world performance, thats real world performance today, but what it would perform like in the future, when GPU's get more powerful, when games use more CPU resource....

Most people actually upgrade thier GPU much more often than the CPU, a lot of people are still on 2500K's / 2600K's with 1060's and 1070's.

I'm on my third GPU with my 4690K, 290, GTX 970 and now GTX 1070, what i'm finding is the 4690K even at 4.5Ghz is starting to struggle, i have enough to buy a 1600 or an i5 and will be keeping it for a few years while upgrading my GPU with every new faster GPU that replaces the 1070 i have now, what do i do?

How much heatroom the CPU has for future proofing is just as important in how it performs today, the only way you can test that is by making the CPU rather than the GPU the driving force behind the frame rates, when you do that the AMD chips out class the Intel ones by a country mile....

Its only fair that readers know that too...
 

4745454b

Titan
Moderator
Edit: actually you hit the nail on the head, yes it matters as well, but the CPU true performance also matters....
The chart you posted is exactly my point

Walk me through it again, because I completely failed to see the point you were trying to make. You said "the Intel CPU is good or better for as long as the GPU performance is not higher" but there is only the one chart showing one GPUs performance. You're talking about info that isn't there and isn't showing what you are claiming.

Edit: You also said in the OP

I think the problem is an assumption that games do not use more than 3 or 4 threads, that is an out of date idea, its an idea that should have been put to rest when you realised that back in the day a 4 core 2500K was always 20% or so faster than the FX-8350, now its often the other way round.

I'd love to see proof of this as well. All the reviews for the 8350 are from 2012 and I don't see any "real" reviews more recent than that. I looked for the 8370 and those were from 2014, is that recent enough? Because you lead with them I'm assuming you are going to post more youtube links which I completely ignore. Just because someone says something on youtube doesn't make them right. Anyone can run a youtube channel, doesn't make them smart or right.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510


Yes there is, can you not see the videos and screen caps i posted?

All the i5's compute threads are at 100% or close enough, while the Ryzen's compute threads are not, with that the performance difference is as much as 60% higher on the Ryzen chip.

The GPU is upo to 60% more bottlenecked on the Intel chip than it is on the AMD one.

 

randomizer

Champion
Moderator


All performance is true performance. What you're referring to is academic performance. That a CPU is fast in a carefully crafted scenario might be nice to know, but if it doesn't reflect the performance under your actual workloads then what good is it for making a purchasing decision?



Nobody how it will perform in the future, nor whether games will use significantly more CPU "resources" in the foreseeable future. I have an i7 920 which could process eight threads back in 2008. It gets butchered by a modern quad core CPU even without SMT. Those extra threads weren't helping back then and they aren't helping now.
 

4745454b

Titan
Moderator
I'm assuming you are going to post more youtube links which I completely ignore. Just because someone says something on youtube doesn't make them right. Anyone can run a youtube channel, doesn't make them smart or right.

Edit: Here is some low res CPU tests. The 7600 has some issues sometimes, but the i7 never loses. (It occurred to me that of the reviews for Ryzen might show the 8350 in newer games.)

https://www.hardocp.com/article/2017/04/11/amd_ryzen_5_1600_1400_cpu_review/4
 

Intrepid_1

Prominent
May 5, 2017
24
0
510


You've just reinforced my point, the more expensive i5 7600K @ 5Ghz loses to the R5 1600 @ 4Ghz.

The i7 costs nearly twice as much.

What HardOCP are also not doing, as you are not doing is publishing the load on the Compute threads, i bit the i5 is at maximum, infact it will be and the i7 not far off.

The one 'game' where the i5 does beat the Ryzen 5 is a 5 year old Bio Shock Infinite, that is one of those ancient games that uses only one compute thread, i know, i have, its terrible even on a clocked up Intel,
With games that are relevant today the i5 is way behind the Ryzen 5.

Actually, again in Metro Last Light the 5Ghz 7700K is only 20% ahead of the 4Ghz Ryzen 5, would you call that a solid victory for Intel? a $350 CPU clocked to 5Ghz is 20% faster than a $220 CPU clocked to 4Ghz.

Personally i think from a cost perspective the Ryzen chip is the clear winner, 20% is not worth $130 more and in everything else we already know the Ryzen chip has the Intel one beat, people do more than just play games, PC's are not consoles. No matter how much you try and claim 'CPU's are just for games' they are not. Gamers upload their gaming to Youtube, more often than not that involves using video editors and encoders.....

On the Ryzen 5 this is their conclusion, its pretty fair....

The AMD Ryzen 5 1600 processor is going to epitomize the "value sweet spot" with its $220 price point. If you are not that person that knows they need a bunch of cores for encoding or rendering, then there is likely no reason for you to buy a 8-Core CPU...yet. However, I can argue that if you want any kind of CPU built for "the future," the 6-Core Ryzen 5 CPU can likely at least deliver you there. I like the Ryzen 5 1600. I like it a lot. AMD is delivering a lot of desktop processing power for the money.

Yeah me too, ^^^^^ it did incredible well against an i7 costing 16GB worth of DDR4 more.... or a really nice Case, a very good NZXT Kraken AIO cooler, perhaps even a 500GB SSD.
Or a GTX 1070 instead of a GTX 1060... hum?

Hats off to them, its not a bad review tho even if a bit content anaemic, i would have liked to have seen a lot more gaming tests relevant to today and some information about how much load the CPU's are under during game testing and how they are coping with it.

Oh and why is he saying the Ryzen 5 CPU is the one for future proofing? is he also realising something you are not?
-----------------

@Randomizer, if the SMT threads are not helping then Intel really are in trouble because the AMD chip with SMT is 60% faster than the Intel Without..... you might like to rethink that.
 

randomizer

Champion
Moderator


SMT didn't help back in 2008 when game engines rarely used more than two threads actively, and isn't helping now because the chip is 9 years old. My point is that purchasing a CPU based on speculated future performance is a waste of time. By the time it makes a difference your CPU is obsolete anyway.
 

4745454b

Titan
Moderator
You've just reinforced my point, the more expensive i5 7600K @ 5Ghz loses to the R5 1600 @ 4Ghz.

While the i5 is slower, I'm not sure "loses" is the word I'd use. In some cases it does win, the synthetic test Unigine and bioshock it wins. By a lot. In the other games the i5 does lose. By a few FPS. Yes the Intel CPUs have more clock speed, but the AMD CPUs do have more cores/threads. You pointed out the one, why is the other ok? Seeing as you like posting quotes from Hard...

This game engine from Oxide is the poster child for scaling according to AMD. However, even with this incredibly thread-aware game engine, the Core i5-7600K only succumbs to the Ryzen 7 while pulling up in a near-dead heat with the Ryzen 5 1600. And while the Intel 7700K wins the race, it is "only" by 14.5%.

It's not like the 7600 is a whipped dog in this race. As a matter of fact that's probably the CPU that made me reply, the 8350. I'd still love to see from a site I trust with nice charts I can review proof that the 8350 in recent games bests the i5 6600 or 7600 all the time.

If I were you I wouldn't get so defensive/mad. You are on the right track somewhat. If someone is testing something they should be testing it, not other things. CPU testing is still one thing where I read the synthetic tests because testing the memory controller or cache hits might matter. If a program I really care about hinges on how well X part of the CPU works. Putting the stress on the CPU for gaming DOES matter. Here is the thing though, keep in mind reviewers don't have a lot of time between getting the parts and writing a 10page+ review. When I did my review here at Toms I think I had around 4-5 days to test AND write up my review. And keep in mind I'm not a paid reviewer for Toms. I still have my normal job to work. When writing they have to give people tests they care about. Tests involving other things like low res or low detail can happen later.

I don't think you can really write ONE review and have it cover everything. That's why I still read Hard. They test CPUs and GPUs in different ways than other review sites and I feel it helps paint a better overall picture of how X performs. Looking at 1080 (or 4K) results or any test that shows what you care about is what matter. I get that you think you've found some fatal flaw in how Toms/we test CPUs. But the fact is gamers play games not at 640 or 720, but 1080 and now soon 4k. And those resolutions are the ones that matter.

I'm not sure why you are complaining about SMT either. Both Intel and AMD have the ability to turn it off, and both cause issues for some things. In Toms review they got better results at times by turning it off. Frankly from your edited out comment about the 290 being faster than the Titan and general put downs of Intel I feel you are really only upset because you are an AMD fanatic. Which makes me sad because I enjoyed looking up data and forming these paragraphs. As someone who won't listen to reason I guess I'll just have to stop.

Edit: I see why you made this thread and at JPs suggestion as well. Looks like I was partly right. If you are that mad at Toms for the "best (gaming) CPU for the money" article there is NOTHING I can say that will change your mind. You are good kid (word, no offense intended.) but I thought you wanted to talk about this. I'll leave you to your ranting. Sorry.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510
I'm trying to keep this civil, i'm not mad, i'm not ranting, i'm simply trying to explain how Toms Hardware could make its reviews more relevant to answering a broader question.
I made it very clear that i wanted "please no hate or bashing" so please. :)

I have said right from the start that i don't disagree with the results, i have also come to agree the methodology is not actually flawed.

At this point there is no need for me to go into much detail about what i'm saying, i have provided plenty of reference material, even you yourself have provided some of it.

To that end 5 year old games designed in an area where dual core CPU's were mainstream are not relevant now.
Using a game that was designed years ago for dual core CPU's as a performance yard stick for games made today for todays CPU's is.... well, it doesn't fulfill that purpose.
If said game designed to use only one or two compute threads and todays games use 8 then the Intel CPU will do better in older games compared with the Ryzen as it has less threads but higher clocked, as a result the i5 is 15% faster in Bio Shock, which really isn't anything to write home about when you consider that the 7600K was clocked at 5Ghz.

A game more relevant to today which makes use of a lot more threads: 'like Metro Last Light' the Ryzen CPU with its huge number of compute threads is 60% faster in that game, yes that is a win for the Ryzen chip, its a huge win.

In that sense your own reviews are actually more relevant than the Bio Shock Infinite slide you keep falling back on, Toms Hardware review game list is more upto date than that.

http://www.tomshardware.com/reviews/amd-ryzen-5-1600x-cpu-review,5014-4.html

Which most of the modern games you tested there using the (real life testing methodology) BF1 / Civ / Dues EX the 7600K and the similar priced 1600K are actually pretty even, other than ROTTR that is..... but thats a whole nother nVidia high CPU software based command thread load balancing overhead argument..... on an RX 480 with Hardware level ASync the results are completely different.

In anycase according to your reviews they are pretty even, yes the 7600K does win out overall but not by a large margin by any stretch of the imagination and the Ryzen 5 is so much better at everything else.... THAT is why a lot of people are very suspicious of your qualifying the "best CPU" as intended for gaming only.
Also you have never actually tested the Ryzen 1500 against the i3 so how can you know the i3 at the same price point is better? i actually doubt it is, certainly the 1500 did very well again the i5's...

My point is i would like to know how these CPU will see me through the future.
The fact of the matter is games will use more CPU resource and GPU's will get more powerfull in the future.
So both games and GPU's will demand more resource from the CPU.

One way you can see how much headroom a CPU has is by making it do more work, you do that by lowering the resolution and the Image Quality.
That method is not some left of field crap, its very much mainstream, reviewers have been doing it for a long time, if i go back through your CPU reviews i bet i'll find some.

So i will repeat the question i and others in the other thread have asked...

If i'm going to buy one of these platforms now, the i5 or the Ryzen 1600 which one is better for future proofing?
The GPU i have have now is a GTX 1070, i plan on keeping the CPU for a few years of GPU upgrades, for naming arguments sake a GTX 1170, a GTX 1270 and possibly a GTX 1370.

In fact i'm going to put it to you like this, given the review material i have referenced in this thread, that when you have enough GPU performance the Ryzen chip has a lot more performance overhead than the i5, so the best platform to go for is the Ryzen 1600, would i be wrong in that assertion?

May i request Toms Hardware investigate this and publish their findings on that.

And can we have a "best CPU" for a more overall performance yard stick, IE not just (exclusively for games)

Many thanks :)
 

PaulAlcorn

Managing Editor: News and Emerging Technology
Editor
Feb 24, 2015
858
315
19,360
Thanks for the feedback, it is appreciated. However, there is one thing that isn't correct.

AdoredTV compared reviews of the FX and -2500K that were conducted over a period of years and claimed the FX got better over time. His video is interesting, but he is comparing game results in reviews that used different hardware and software platforms over time. That is an inherent flaw that can exaggerate performance deltas. His findings spurred other Youtubers to go back and actually test with static test platforms. Check out some of the videos where other Youtubers fact-checked the analysis, but instead, used static tests systems.

https://www.youtube.com/watch?v=76-8-4qcpPo

As you can see, the 2500K beats the -8370 by ~20% in average frame rates (and ~12% in minimums). Or, as he put it, "manhandled" the FX. This was with 16 modern games, mind you. If he did frame time analysis and checked 99th percentile frametimes and frame variance, I bet the picture would be far worse than 20%.

We all want games to scale better over cores, but that hasn't happened yet. These same things were said back in the past, and it still hasn't happened. That's not to say that there isn't some value to the argument, but I personally don't think its wise to recommend products based on possibilities or things that might happen.

Also, those tests are conducted against a stock -6600K, a previous-gen chip. Now, if we compared a modern -7600K OC'd to 5.0GHz, which is very simple to do, against the Ryzen's max 4.0GHz (I rarely rarely see anyone hit 4.1) the story would be different.

Also, dialing back the resolution and quality essentially creates a synthetic test, and as we see from synthetic tests, those don't often equate well to the real world.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510


Thanks for the reply :)

Again i'm not disputing your results, and thats not just because Ryzen 5 @ 4Ghz is not actually far behind the 7600K at 5Ghz

I added all the scores (but decimal points) up to see how far behind is actually is overall, the scores are 1075 to the 7600K and 979 to the 1600X, If the 1600X is at 100% the 7600K is at 111%, so 11%, i don't think that's significant, its actually only down to maybe 3 games like GTA-V and Tomb Raider, the latter running on an AMD card has = performance to Intel, its actually down to nVidia's software thread scheduling overheads vs AMD's Hardware scheduling.... i'm sure nVidia will sort it out with driver as some point, they usually do.

As for the FX-8350 performance, like anything it depends on the game, even what part of the same game....

This has always been my problem with slides without context.
Here are a couple more slides...

Before i do, personally. He is not someone i have any respect for, i distrust him, i think i'll just not bother commenting on his reviews.

Anyway... those slides....

CPU_01_zps1c17313e.png


http--wwwgamegpuru-images-stories-Test_GPU-Action-Battlefield_4-test-bf4_proz_2_zps72229e76.jpg


RhKu6bm_zps171c6672.jpg


2868723-9170151442-http-_zpsnpy1rn3m.jpg.html


There you go, an FX-8350 beating not just the i5 but sometimes the i7.

Look, :) there is a lot more i want to say but its very late for me... i may comeback to this soon... :)

PS: speaking of AdoredTV, his channel is one i like a lot, and it is becoming very very respected amongst enthusiasts and everyday consumers. he knows what he is doing, he knows what he is talking about.
I think the point he makes is spot on.

As a hobby i am very familiar with game engines and development, i do recognise what he is talking about, games are becoming more CPU resource intensive and its resource Intel increasingly don't have, if you did publish the Compute thread loads of your reviews i'm sure that would be very clear.
 

Intrepid_1

Prominent
May 5, 2017
24
0
510
Thinking more about this i went back to look at some more results.
You never actually made a Ryzen vs i3 review, but you did compare the 7350K to the 7600K and the 1500X to the 7600K.

Ryzen 1500X Review
http://www.tomshardware.co.uk/amd-ryzen-5-1500x-cpu,review-33880.html

i3 7350 Review
http://www.tomshardware.co.uk/intel-core-i3-7350k,review-33797-2.html

The 7600K is 12% faster overall than the 1500X.

The 7600K is 15% faster overall than the 7350K

So the 7600K is faster than the 7350K more so than the 7600K is to the Ryzen 1500X, IE the 4 core Ryzen is faster than the 2 core i3. In games according to your own reviews.

Not by much but none the less faster, the 1500X on Amazon USA is $180 while the i3 7350 is $170, the Ryzen 1400 (which is exactly the same chip as the 1500X just lower clocked out of the box) is $160.

To reiterate the 7600K is 12% faster than the 1500X, the 7600K is $220 while the 1500X is $40 less and the 1400 a whopping $60 less.

I realise the ones you have as "the Best CPU" in the sub $200 range are the i3 7100 and the i5 7500.

So i looked at them, the i5 7500 is 10% faster than the 1500X but $30 more expensive, or $50 vs the 1400

I couldn't find your review for the i3 7100 but its clocked a lot lower than the 7350K and it looks to me like the 1500X beats that, marginally.

Even ignoring the price for performance ratio excluding everything but games, which in its self is not true to real life at all as very few actually use their PC's for nothing but gaming, and the fact that none of your sub $200 Intel chips are unlock while all the AMD ones are, ignoring that, some of those AMD chip i mentioned are still a beter Chip. and yet not even an honourable mention for any AMD chip? you gave Intel a total and utter blue wash?
For me the Ryzen 1400 is better for its money than the locked i3 and the locked i5.
 

Supahos

Expert
Ambassador
First a 1400 and 1500x are not identical, check cache numbers before you make the claim. Admittedly 8mb is usually enough not to hurt the performance in gaming. Also the 7600k test beds weren't the same in both tests so to try to extrapolate data from the two tests isn't real accurate. Not saying that an i3 is a better option than any of the chips, just that basically every point you made is invalid.
 
Status
Not open for further replies.

Latest posts