Sign in with
Sign up | Sign in
Your question
Closed

Samsung Accused of Manipulating Benchmarks Again

Last response: in News comments
Share
October 2, 2013 4:57:42 AM

I`m confused, so... technically Samsung is "cheating" because they allow their CPU to run at full speed sometimes? ... that`s a "shocker". And i still don`t get it why would that CPU idle 3 cores when running a benchmark .. isn`t that application suppose to max out the hardware?
Score
-6
October 2, 2013 5:03:21 AM

Samsung is "cheating" because they purposefully created a bit a code that checks for the identity of the app that is running. That bit of code has a hard coded list of names that it checks to see if the app that is running is a popular benchmarking app. When it detects that it is, it behaves differently by going full throttle on its cpu core speeds to affect the outcome of the benchmark scores even before the benchmark is running.

This is disingenuous since the performance seen on the benchmark is not truly what you can expect in real-world situations. Instead, the score has been overinflated thanks to that bit of code detecting the benchmarking app, thereby making people think that the device is more powerful than it truly is in real-world situations.
Score
13
Related resources
October 2, 2013 5:08:21 AM

In short next time you benchmark phones make sure to modify the benchmark name first to get a fair score.
Score
9
October 2, 2013 5:15:38 AM

Well then.......Intel is cheating all the time with turboboost and Hyperthreading
Score
-13
October 2, 2013 5:40:55 AM

so "benchmark mode" is where the processor runs at maximum and shows what the phone can do when on full belt
and that's cheating? I thought a benchmark is to see what the device can do when pushed
Score
-11
October 2, 2013 5:41:07 AM

If you don't understand the basics better not write anything here. First it cheats because the phone in real life does not work like that. I hope you really understand that, it does not work like that in real life. If they do that at least they should say, but I am very curious why they don't do that with a game???? Because the battery which is very bad will be even worst maybe? Second the SoC on this phone the Snapdragon 800 has the same features as Turbo Boost, HyperThreading is not available here! Except that with Intel they don't need to keep all the cores online and at full speed when the pc is doing nothing just rendering the interface of a benchmark app. They should copy Intel and allow this boost of frequency on the fly but it seems for whatever reason if they do that the values are lower so that's the reason why they turn the CPU on and keep the cores running at 2,3Ghz doing nothing. Either way a cheap brand, cheap products(quality wise), cheap customization and add the regional lock to that and a high price. But there are people stupid enough to call this a premium device instead of a big scream device with average performance and a cartoonish OS. Don't get me wrong I don't hate Android as I own both IOS and Android but some things need t be said the right way.
Score
-6
October 2, 2013 5:42:43 AM

Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing off
Score
5
October 2, 2013 5:44:59 AM

Thats the keyword, all the time. Its ok to turboboost to any real application all the time as u benefit from it but it is cheating if u only turbo boost on benchmark for the sake of showing off
Score
2
October 2, 2013 5:46:13 AM

Idiotic move on Samsung's part. The phone is plenty fast without crap like this.
Esp. since benchmarks now matter to only the few that still place any value into them (I am referring to phones benchmarks). For daily use they're totally irrelevant, people could give a rat's a$$ about bench results, and all is running the way it's supposed to run. Why risk getting caught like this, when you know the OS is open and can (and will) be examined by everybody with a dog in it?
Score
2
October 2, 2013 5:57:58 AM

Muhibah said:
Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing off

I'm not defending what's been done, but that's not the case here. Turbo is generally available to any application, and is a low-level CPU load and TDP-dependent operating; not a software/code-dependent thing that only allows turbo under very specific conditions and applications. Manipulating CPU response in the way that Samsung is allegedly doing just makes benchmark comparisons fairly worthless--even more than they already were for extracting real-world usage experience.
Score
5
October 2, 2013 6:04:13 AM

What do you think the reason are for that, besides the higher score?
Is it a temp thing? Battery consumption? Im talking about the reason they dont use all cores 24/7.
Score
-2
October 2, 2013 6:10:22 AM

Shocked at the sheer stupidity of people who don't see what's wrong with this or support Samsung. And I thought Apple fanboys where bad!
Score
1
October 2, 2013 6:31:57 AM

Well... we still have to wait a good amount of time till smartphones actually do things that are nice (like playing crysis), so I guess a few bench here or there aint gonna make much of a difference unless the OS is programed in a similar way.
Score
-4
October 2, 2013 6:40:26 AM

Next we will have people complaining about me setting my pc bios to my maximum overclock when running 3dmark, because I don't run my OC like that 24/7?
Score
-7
October 2, 2013 6:40:54 AM

Do people really care that Samsung gave the benchmark score for the speeds their processors are capable of? Can't any app use full processing power? Am I wrong? I think Samsung can use any clock rate that they sell the phone with. Overclocking and THEN bench marking would be cheating IMHO.
Score
-2
October 2, 2013 6:47:34 AM

I really don't understand how people think this isn't cheating... It's as if you have this great car that unlocks all of its horsepower and performance when a car reviewer decides to drive it, but then dumbs everything down to prius levels of drivability when anyone else drives it.

Sure this phone certainly has the potential to run great with all 4 cores maxed out but just like the article stated, what's the point if it'll never run like that for any other piece of software you decide to use that isn't a benchmark?
Score
14
October 2, 2013 6:52:03 AM

Isn't it common sense to disable power saving features when using synthetic benchmarking?
Score
-5
October 2, 2013 6:57:23 AM

Isn't it common sense to disable power saving features when using synthetic benchmarking?
Score
-5
October 2, 2013 7:00:05 AM

damric said:
Isn't it common sense to disable power saving features when using synthetic benchmarking?


Are other phone companies doing it when they benchmark their phones? If not, then it's only "common sense" in that you're painfully aware you're using a different and advantageous set of rules compared to your competitors and trying to pawn it off as if you're playing by the same rules. Sounds like cheating to me.

I can understand this practice for some hypothetical "how powerful COULD this phone be"... But when you buy a phone that says "This phone is X powerful" but it will never be more than 80% of X when you actually use it because of some benchmarking trick like this, that's disingenuous.

It's not much different than the Surface RT having a 32gb SSD. Yes, in theory, absolutely true. When 14 or so gb of that is taken up prior to you even turning the device on for the first time, advertising 32gb of storage without a notable footnote is pretty disingenuous. Go figure, people were *flipping out* when MS misrepresented the storage space on their tablets like that...
Score
6
October 2, 2013 7:07:47 AM

Quote:
In short next time you benchmark phones make sure to modify the benchmark name first to get a fair score.

Next time you test battery life, make sure to use these benchmarks to test it.
Score
4
October 2, 2013 7:08:45 AM

I'm not sure how the note 3 run's, but with my ROOTED tf101 I can force it to stay maxed. Now, this phone is not rooted and it's "out of the box testing" so that point might be a little moot. However I must agree with some other folks, were bench marking the CPU, not the "phone". If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though and I believe it's being way over hyped.

Guy 1: Yo, I benchmarked my intel core I7, it did a score of over 9000
Guy 2: Yeah, but you used the processor full potential, so it don't count cause it does not run at 5ghz when you click the start menu.
Score
-2
October 2, 2013 7:09:11 AM

I'm not sure how the note 3 run's, but with my ROOTED tf101 I can force it to stay maxed. Now, this phone is not rooted and it's "out of the box testing" so that point might be a little moot. However I must agree with some other folks, were bench marking the CPU, not the "phone". If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though and I believe it's being way over hyped.

Guy 1: Yo, I benchmarked my intel core I7, it did a score of over 9000
Guy 2: Yeah, but you used the processor full potential, so it don't count cause it does not run at 5ghz when you click the start menu.
Score
-4
October 2, 2013 7:11:40 AM

I see it as a tool to mod and add my own apps onto this list and make those apps run faster. I'll probably need a charger handy though.
Score
1
October 2, 2013 7:15:53 AM

This is because people care too much about the synthetic benchmarks, and they shouldn't, those benchies mean nothing to real-life usage
Score
-2
October 2, 2013 7:24:41 AM

kristi_metal said:
This is because people care too much about the synthetic benchmarks, and they shouldn't, those benchies mean nothing to real-life usage


Usually true. If you add in that the benchmark program artificially overclocks the processor in a way that no other program does and it's extra true.
Score
0
October 2, 2013 7:29:05 AM

@haswell5271 short answers: 1. yes. 2. no. 3. yes. long answers: 1. yes, of course people care, why wouldn't they want to know the performance of whatever product they're buying? 2. no, that's the whole point of this article. there is code written into the OS that ONLY when benchmarking software is detected will the CPU/GPU run at full speed. every other bit of software will run at a throttled down speed, that's why it's CHEATING THE BENCHMARK. 3. yes, you are wrong, for reasons why you're wrong and why the rest of your post misses the point completely, please see above answers and reread the article.
Score
2
October 2, 2013 7:37:53 AM

@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
Score
3
October 2, 2013 7:39:57 AM

@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
Score
0
October 2, 2013 7:44:45 AM

@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
Score
0
October 2, 2013 8:14:22 AM

I have no tide with Samsung and not owing any of their latest hi-end phones, but honestly, how many of smartphones/notes buyers really care about benchmarking. For me, I am more interested in how good the screen is, how long the battery can last and how high the resolution of the camera can go.
Score
0
October 2, 2013 8:23:53 AM

Quote:
justhandguns
I have no tide with Samsung and not owing any of their latest hi-end phones, but honestly, how many of smartphones/notes buyers really care about benchmarking. For me, I am more interested in how good the screen is, how long the battery can last and how high the resolution of the camera can go.


Agreed, so why would Samsung try to cheat the results?

To me, these companies know most customers don't look at benchmarks but they do know if they can advertise, "*Fastest phone!"

* based on highest score of cpu on tests X,Y,Z

there's a marketing advantage, so I feel the cheating is not for the people looking at benchmarks because most of them will be aware of Samsung's shameless tactics. Its more for some promotion / marketing at the highest.
Score
-1
October 2, 2013 8:43:23 AM

Disingenuous said:
Samsung is "cheating" because they purposefully created a bit a code that checks for the identity of the app that is running. That bit of code has a hard coded list of names that it checks to see if the app that is running is a popular benchmarking app. When it detects that it is, it behaves differently by going full throttle on its cpu core speeds to affect the outcome of the benchmark scores even before the benchmark is running.

This is disingenuous since the performance seen on the benchmark is not truly what you can expect in real-world situations. Instead, the score has been overinflated thanks to that bit of code detecting the benchmarking app, thereby making people think that the device is more powerful than it truly is in real-world situations.


And you think benchmarks show the true performance of hardware in the real world anyways?

So when a 8 core CPU gets 2x the score of a quad core, the majority of real world situations would call for that same 2x scaling, correct?

The problem is that benchmarks are just benchmarks. To truly test it you want to run real apps and see the performance for yourself. I have a S4 and I can tell you that it is truly a powerful phone that's smooth compared to my Droid Razr Maxx work phone.

Most of the Android smartphones will use the same CPU in the same market segment. The top end is on the Snapdragon 800 for the most a part and apart from difference in clock speed they are the same at the core level so overall performance will be the exact same in the real world.

I don't like the idea of inflated benchmarks but then again if you are using synthetic benchmarks as a gauge for real world performance and not for what it is, which is performance in a perfect world then you need to rethink benchmarks.
Score
-2
October 2, 2013 8:52:14 AM

@calam1ty - there are massive gas guzzling engines that shut off cylinders to *save fuel*, I bet they state the max horse power not the hp while fuel saving is active.

I don't really see the problem with this to be honest. To me a benchmark should be the maximum attainable performance for the system being tested, under normal circumstances. So for instance, if an app CAN use all four cores at 2.3ghz then to me thats what I want benchmarked.

That being said I don't see why the benchmark doesn't stress the phone enough to automatically enable the other cores?
Also there's only a 20% gain by forcing full power mode? Thats 4x 2.3ghz versus 1x 300Mhz?
Score
0
October 2, 2013 9:03:50 AM

@Iatemyself you missed the point completely, the note 3 WILL NOT run at 2.6Ghz, stock is 2.4ghz. Hence why I used that specific frequency if it ran at +200mhz that would be a pure overclock for one reason.

Sorry if it was confusing.
Score
-1
October 2, 2013 9:07:58 AM

Sorry but This is not cheating what it is doing is showing what the phone can do. There is nothing stopping other companies from doing the same. I myslef when benchmark put my Xperia ZL into full throttle to see what my max scores will be. From what I am seeing here from people is they would rather see lower scores from a given device which does not relate to what the hardware can actually do. The whole idea of a benchmark is to get the most out of the hardware so no samsumg has done nothing wrong here all it means is no one else had the idea first.

If you are going to give samsung backlash for this well then maybe you should do the same for pretty much every hardware site on the internet because they all overclock the hardware to get the most out of it. Some will even put a overclocked video card against a factory clocked card & say the factory card is slower ummm will no duh.

It is not like samsung actually overclocked their hardware it is all running within spec but set to performance mode which most of us do anyways when we bench our own device. So to expect anything less from the hardware when I bench a device is just insane. It would be like getting a car hyrid that has a v6 in it but only runs on 3 while everyday driving. Let me tell you I would be upset if I stepped on the pedal & all six pistons did not produce power. Same idea so what people are saying is samsung is wrong for turning it up a notch when benchmarking but these same people would be crying how slow the device was if they left it in battery mode or if samsung left it in performance mode all the time they would complain how their battery life sucked. I guess you can not please everyone. By the way I am not a samsung fanboi I dont own any phone that is samsung because I don't like the look of a galaxy device.
Score
-1
October 2, 2013 9:20:59 AM

@hardrock40 "Sorry but This is not cheating what it is doing is showing what the phone can do."

this is showing what the phone can do only under specific circumstances as defined by samsung, which is, when it is detecting a certain benchmark. the article stated that taking the same benchmark and renaming it, the performance enhancement did not engage, simply because the coded file samsung put onto the phone did not recognize the name of the benchmark. this is not "performance mode" vs "battery mode". to fit into your analogy with the v6 in a hybrid... the equivalent of what samsung did is all 6 cylinders are running when flooring it on the daytona speedway, but then when trying to do the same on an unknown track it automatically shuts off 3 cylinders.
Score
0
October 2, 2013 9:21:22 AM

Quote:
hardrock40
Sorry but This is not cheating what it is doing is showing what the phone can do. There is nothing stopping other companies from doing the same. I myslef when benchmark put my Xperia ZL into full throttle to see what my max scores will be. From what I am seeing here from people is they would rather see lower scores from a given device which does not relate to what the hardware can actually do. The whole idea of a benchmark is to get the most out of the hardware so no samsumg has done nothing wrong here all it means is no one else had the idea first.

If you are going to give samsung backlash for this well then maybe you should do the same for pretty much every hardware site on the internet because they all overclock the hardware to get the most out of it. Some will even put a overclocked video card against a factory clocked card & say the factory card is slower ummm will no duh.

It is not like samsung actually overclocked their hardware it is all running within spec but set to performance mode which most of us do anyways when we bench our own device. So to expect anything less from the hardware when I bench a device is just insane. It would be like getting a car hyrid that has a v6 in it but only runs on 3 while everyday driving. Let me tell you I would be upset if I stepped on the pedal & all six pistons did not produce power. Same idea so what people are saying is samsung is wrong for turning it up a notch when benchmarking but these same people would be crying how slow the device was if they left it in battery mode or if samsung left it in performance mode all the time they would complain how their battery life sucked. I guess you can not please everyone. By the way I am not a samsung fanboi I dont own any phone that is samsung because I don't like the look of a galaxy device.


No, its like a car having a special key for auto reviewers/testers. When they insert the key the car gets an extra 20% Horse Power unlocked and augments the test results.

Whereas another car actually using the same/similar engine, didn't have a special key for reviewers/testers and their results are actually what everyday consumers would get.

You cannot replicate benchmark results with other apps since Samsung treats benchmark apps different than other apps even though they are as resource intensive.

Score
0
October 2, 2013 9:30:54 AM

So Samsung made benchmarks run good on its phone. I ask the simple question of "How does this impact you?" Is the phone slower than you anticipated? Does this impact your decision to buy one or not? Samsung is forcing their hardware to run full throttle when benchmarking.

That doesn't sound like cheating to me. What it sounds like is everyone else complaining because their phone doesn't have code to specifically run at max speed when its running a benchmark. Weather or not this is indicative or "real world" app performance, it is indicative of "maximum" performance.

P.S. - cheating on a benchmark suite is historically all they are good for imo.
Score
-1
October 2, 2013 9:31:08 AM

double post bug strikes again
Score
-3
October 2, 2013 9:40:15 AM

Glad I went with z1
Score
0
October 2, 2013 9:43:02 AM

it is kind of silly that they did this but in the end the phone really did as well as the benchmark said it did. inflating numbers makes me think they lied about performance which they did not.

Either way you look at it the phone did as well as the bench said it did.
Score
1
October 2, 2013 9:46:31 AM

Yeah, phone benchmark are so important. /sarcasm

Are benchmark scores part of the criteria folks are using to compare and purchase phones nowadays?
Score
1
October 2, 2013 9:49:33 AM

halcyon said:
Yeah, phone benchmark are so important. /sarcasm

Are benchmark scores part of the criteria folks are using to compare and purchase phones nowadays?


You know, since a lot of people game on phones, there are actually a lot who place stock on how powerful the hardware is in a phone. When someone is going in to buy a phone saying "I want the fastest phone for my buck" and Samsung has skewed benchmark results compared to the competitions, this is a potentially decision making factor. It amazes me how fast so many people are to excuse this type of BS.
Score
0
October 2, 2013 9:51:11 AM

Well the point is would you want your device to be running full bore at all times & draining the battery faster no you would not. Did samsung do anything wrong not really all they did was make sure the device was able to perform at full speed when the user is testing their device to see how fast it is. This is called performance profiling and is not meant to be used for everyday use but just when benchmarking.

Did they make it a bit hinking maybe but I prefer knowing that my device is set to max when testing for speed. By the way why wouldn't they use a profile made for the benchmarks how else is it supposed to know when a user is running a set of tests. AMD & Nvidia have performance profiles for their Video cards that tweak setting for different games to make them run faster or tweaks for the benchmarks themselves but you rarely hear about that on these sites. Why because it is just common sense that a device maker or card maker will want to make sure their hardware is giving the best result so they give a better user experience to the end user.

like I said samsung & most likely others to follow will be doing this type of thing for years to come & I for one want my device to perform at it's best when testing it so I don't feel cheated by a purchase I just made. It is called using performance profiling for benchmarks it has been going on in the PC world for years & now a mobile device maker is doing it & everyone thinks it is a bad thing. just my 2 cents on this & just my opinion on it. Everyone has the right to have their own opinion in this world.
Score
2
October 2, 2013 10:10:56 AM

Just amazing how far Scamsung will go to beat fellow Android companies and Apple. Samsung does it fact make quality products but its business ethics are extremely poor. I liked Samsung until I read all about their business practices in Korea and abroad.
Score
0
October 2, 2013 10:20:30 AM

If benchmarks were irrelevant than why would Samsung do this?

Just because YOU don't use them doesn't meant everyone doesn't.

Just because YOU don't buy a car based on performance results doesn't mean everyone else does the same as you.

Score
-2
October 2, 2013 10:21:25 AM

@hardrock40 i'm not sure how you still don't understand what happened here, but based on your response you really have no idea what the article is saying and missed the point completely. perhaps you should consider reading the article over again.
Score
-1
October 2, 2013 10:31:01 AM

And here I thought Overclocking was all about finding the maximum your PC would run stable 24/7. Atleast that's what I do. My 980x purrs along at 4.6Ghz happily all day long. I do have Watercooling, but still. It's my stable max 24/7 speed.
Score
0
October 2, 2013 10:51:09 AM

"Shocked at the sheer stupidity of people who don't see what's wrong with this or support Samsung. And I thought Apple fanboys where bad!"
I own an S3 and don`t really care about benchmarks,since most of the current generation phones are pretty close in performance, in fact never installed one in my phone, i bought it because i like it better than iphone 4/5, so pretty much i don`t give a s*** if apple/htc/samsung/lg/sony does anything similar.
Score
0
!