Sign in with
Sign up | Sign in

Samsung Accused of Manipulating Benchmarks Again

By - Source: Ars Technica | B 71 comments

Ars Technica presents its evidence against Samsung.

Samsung is again accused of manipulating benchmarks, this time so that the Galaxy Note 3 scores really well. The company was previously accused of inflating Galaxy S4 benchmarks earlier this summer, as the GPU and CPU clock speeds would change depending on the benchmark used. Now Ars Technica has discovered another spec inflation with this latest Note 3 phone.

The site claims that the 2.3 GHz Snapdragon 800 chip used in Samsung's Galaxy Note 3 phone highly outperforms the same chip used in the similarly specced LG G2 phone. After a little investigating, the site discovered that Samsung is supposedly artificially boosting the U.S. Note 3's scores with a special high-power CPU mode that kicks in when a large number of popular benchmarks are running.

According to the report, when the Note 3 is idling normally, three of the four cores are turned off while the fourth is idled down to 300 MHz. However when running a benchmark app, all four cores are forced to remain active and maxed out at 2.3 GHz – they are not allowed to shut off. Even though idling has nothing to do with benchmarking scores, this was the first sign of foul play, as a device shouldn't treat a benchmarking app differently than a normal app.

Ars Technica then discovered that the phone goes into benchmarking mode when the CPU detects specific benchmarking apps. For instance, the site disassembled the Geekbench 3 app, renamed the package as Stealthbench 3, and then reassembled it. The Note 3 went into benchmark mode when Geekbench was loaded, but not when the renamed app was loaded.

Geekbench also showed that in System Monitor, all four cores were active and maxed out at 2.3 GHz, whereas the renamed app revealed that the same chip turned off three cores and idled down the fourth to 300 MHz. Additional testing revealed that the Note 3's benchmarking mode gives the device a 20 percent boost over its natural score, which is similar to the LG G2 phone, if not slightly better.

The benchmark mode is triggered by a file named "DVFSHelper.java" that contains a hard-coded list of every package that is affected by the CPU boosting. The "DVFS" in "DVFSHelper" actually stands for "Dynamic frequency scaling", also known as CPU throttling, the site reports. This DVFSHelper function is used exclusively for benchmarks, Ars claims, covering all the popular benchmarking apps including Geekbench, Quadrant, Antutu, Linpack, GFXBench and several Samsung benchmarks.

Ars Technica continues to plead its case against Samsung here in this report.

Discuss
Ask a Category Expert

Create a new thread in the News comments forum about this subject

Example: Notebook, Android, SSD hard drive

This thread is closed for comments
Top Comments
  • 14 Hide
    calam1ty , October 2, 2013 6:47 AM
    I really don't understand how people think this isn't cheating... It's as if you have this great car that unlocks all of its horsepower and performance when a car reviewer decides to drive it, but then dumbs everything down to prius levels of drivability when anyone else drives it.

    Sure this phone certainly has the potential to run great with all 4 cores maxed out but just like the article stated, what's the point if it'll never run like that for any other piece of software you decide to use that isn't a benchmark?
  • 13 Hide
    Disingenuous , October 2, 2013 5:03 AM
    Samsung is "cheating" because they purposefully created a bit a code that checks for the identity of the app that is running. That bit of code has a hard coded list of names that it checks to see if the app that is running is a popular benchmarking app. When it detects that it is, it behaves differently by going full throttle on its cpu core speeds to affect the outcome of the benchmark scores even before the benchmark is running.

    This is disingenuous since the performance seen on the benchmark is not truly what you can expect in real-world situations. Instead, the score has been overinflated thanks to that bit of code detecting the benchmarking app, thereby making people think that the device is more powerful than it truly is in real-world situations.
Other Comments
  • -6 Hide
    ohim , October 2, 2013 4:57 AM
    I`m confused, so... technically Samsung is "cheating" because they allow their CPU to run at full speed sometimes? ... that`s a "shocker". And i still don`t get it why would that CPU idle 3 cores when running a benchmark .. isn`t that application suppose to max out the hardware?
  • Display all 71 comments.
  • 13 Hide
    Disingenuous , October 2, 2013 5:03 AM
    Samsung is "cheating" because they purposefully created a bit a code that checks for the identity of the app that is running. That bit of code has a hard coded list of names that it checks to see if the app that is running is a popular benchmarking app. When it detects that it is, it behaves differently by going full throttle on its cpu core speeds to affect the outcome of the benchmark scores even before the benchmark is running.

    This is disingenuous since the performance seen on the benchmark is not truly what you can expect in real-world situations. Instead, the score has been overinflated thanks to that bit of code detecting the benchmarking app, thereby making people think that the device is more powerful than it truly is in real-world situations.
  • 9 Hide
    qlum , October 2, 2013 5:08 AM
    In short next time you benchmark phones make sure to modify the benchmark name first to get a fair score.
  • -6 Hide
    laseru , October 2, 2013 5:41 AM
    If you don't understand the basics better not write anything here. First it cheats because the phone in real life does not work like that. I hope you really understand that, it does not work like that in real life. If they do that at least they should say, but I am very curious why they don't do that with a game???? Because the battery which is very bad will be even worst maybe? Second the SoC on this phone the Snapdragon 800 has the same features as Turbo Boost, HyperThreading is not available here! Except that with Intel they don't need to keep all the cores online and at full speed when the pc is doing nothing just rendering the interface of a benchmark app. They should copy Intel and allow this boost of frequency on the fly but it seems for whatever reason if they do that the values are lower so that's the reason why they turn the CPU on and keep the cores running at 2,3Ghz doing nothing. Either way a cheap brand, cheap products(quality wise), cheap customization and add the regional lock to that and a high price. But there are people stupid enough to call this a premium device instead of a big scream device with average performance and a cartoonish OS. Don't get me wrong I don't hate Android as I own both IOS and Android but some things need t be said the right way.
  • 5 Hide
    Muhibah , October 2, 2013 5:42 AM
    Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing off
  • 2 Hide
    Muhibah , October 2, 2013 5:44 AM
    Thats the keyword, all the time. Its ok to turboboost to any real application all the time as u benefit from it but it is cheating if u only turbo boost on benchmark for the sake of showing off
  • 2 Hide
    house70 , October 2, 2013 5:46 AM
    Idiotic move on Samsung's part. The phone is plenty fast without crap like this.
    Esp. since benchmarks now matter to only the few that still place any value into them (I am referring to phones benchmarks). For daily use they're totally irrelevant, people could give a rat's a$$ about bench results, and all is running the way it's supposed to run. Why risk getting caught like this, when you know the OS is open and can (and will) be examined by everybody with a dog in it?
  • 5 Hide
    teh_chem , October 2, 2013 5:57 AM
    Quote:
    Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing off

    I'm not defending what's been done, but that's not the case here. Turbo is generally available to any application, and is a low-level CPU load and TDP-dependent operating; not a software/code-dependent thing that only allows turbo under very specific conditions and applications. Manipulating CPU response in the way that Samsung is allegedly doing just makes benchmark comparisons fairly worthless--even more than they already were for extracting real-world usage experience.
  • -2 Hide
    Lord Captivus , October 2, 2013 6:04 AM
    What do you think the reason are for that, besides the higher score?
    Is it a temp thing? Battery consumption? Im talking about the reason they dont use all cores 24/7.
  • 1 Hide
    wemakeourfuture , October 2, 2013 6:10 AM
    Shocked at the sheer stupidity of people who don't see what's wrong with this or support Samsung. And I thought Apple fanboys where bad!
  • -4 Hide
    cats_Paw , October 2, 2013 6:31 AM
    Well... we still have to wait a good amount of time till smartphones actually do things that are nice (like playing crysis), so I guess a few bench here or there aint gonna make much of a difference unless the OS is programed in a similar way.
  • -7 Hide
    damric , October 2, 2013 6:40 AM
    Next we will have people complaining about me setting my pc bios to my maximum overclock when running 3dmark, because I don't run my OC like that 24/7?
  • -2 Hide
    haswell5271 , October 2, 2013 6:40 AM
    Do people really care that Samsung gave the benchmark score for the speeds their processors are capable of? Can't any app use full processing power? Am I wrong? I think Samsung can use any clock rate that they sell the phone with. Overclocking and THEN bench marking would be cheating IMHO.
  • 14 Hide
    calam1ty , October 2, 2013 6:47 AM
    I really don't understand how people think this isn't cheating... It's as if you have this great car that unlocks all of its horsepower and performance when a car reviewer decides to drive it, but then dumbs everything down to prius levels of drivability when anyone else drives it.

    Sure this phone certainly has the potential to run great with all 4 cores maxed out but just like the article stated, what's the point if it'll never run like that for any other piece of software you decide to use that isn't a benchmark?
  • -5 Hide
    damric , October 2, 2013 6:52 AM
    Isn't it common sense to disable power saving features when using synthetic benchmarking?
  • -5 Hide
    damric , October 2, 2013 6:57 AM
    Isn't it common sense to disable power saving features when using synthetic benchmarking?
  • 6 Hide
    stevejnb , October 2, 2013 7:00 AM
    Quote:
    Isn't it common sense to disable power saving features when using synthetic benchmarking?


    Are other phone companies doing it when they benchmark their phones? If not, then it's only "common sense" in that you're painfully aware you're using a different and advantageous set of rules compared to your competitors and trying to pawn it off as if you're playing by the same rules. Sounds like cheating to me.

    I can understand this practice for some hypothetical "how powerful COULD this phone be"... But when you buy a phone that says "This phone is X powerful" but it will never be more than 80% of X when you actually use it because of some benchmarking trick like this, that's disingenuous.

    It's not much different than the Surface RT having a 32gb SSD. Yes, in theory, absolutely true. When 14 or so gb of that is taken up prior to you even turning the device on for the first time, advertising 32gb of storage without a notable footnote is pretty disingenuous. Go figure, people were *flipping out* when MS misrepresented the storage space on their tablets like that...
  • 4 Hide
    Bloob , October 2, 2013 7:07 AM
    Quote:
    In short next time you benchmark phones make sure to modify the benchmark name first to get a fair score.

    Next time you test battery life, make sure to use these benchmarks to test it.
Display more comments