Samsung Accused of Manipulating Benchmarks Again
Ars Technica presents its evidence against Samsung.
Samsung is again accused of manipulating benchmarks, this time so that the Galaxy Note 3 scores really well. The company was previously accused of inflating Galaxy S4 benchmarks earlier this summer, as the GPU and CPU clock speeds would change depending on the benchmark used. Now Ars Technica has discovered another spec inflation with this latest Note 3 phone.
The site claims that the 2.3 GHz Snapdragon 800 chip used in Samsung's Galaxy Note 3 phone highly outperforms the same chip used in the similarly specced LG G2 phone. After a little investigating, the site discovered that Samsung is supposedly artificially boosting the U.S. Note 3's scores with a special high-power CPU mode that kicks in when a large number of popular benchmarks are running.
According to the report, when the Note 3 is idling normally, three of the four cores are turned off while the fourth is idled down to 300 MHz. However when running a benchmark app, all four cores are forced to remain active and maxed out at 2.3 GHz – they are not allowed to shut off. Even though idling has nothing to do with benchmarking scores, this was the first sign of foul play, as a device shouldn't treat a benchmarking app differently than a normal app.
Ars Technica then discovered that the phone goes into benchmarking mode when the CPU detects specific benchmarking apps. For instance, the site disassembled the Geekbench 3 app, renamed the package as Stealthbench 3, and then reassembled it. The Note 3 went into benchmark mode when Geekbench was loaded, but not when the renamed app was loaded.
Geekbench also showed that in System Monitor, all four cores were active and maxed out at 2.3 GHz, whereas the renamed app revealed that the same chip turned off three cores and idled down the fourth to 300 MHz. Additional testing revealed that the Note 3's benchmarking mode gives the device a 20 percent boost over its natural score, which is similar to the LG G2 phone, if not slightly better.
The benchmark mode is triggered by a file named "DVFSHelper.java" that contains a hard-coded list of every package that is affected by the CPU boosting. The "DVFS" in "DVFSHelper" actually stands for "Dynamic frequency scaling", also known as CPU throttling, the site reports. This DVFSHelper function is used exclusively for benchmarks, Ars claims, covering all the popular benchmarking apps including Geekbench, Quadrant, Antutu, Linpack, GFXBench and several Samsung benchmarks.
Ars Technica continues to plead its case against Samsung here in this report.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
Kevin Parrish has over a decade of experience as a writer, editor, and product tester. His work focused on computer hardware, networking equipment, smartphones, tablets, gaming consoles, and other internet-connected devices. His work has appeared in Tom's Hardware, Tom's Guide, Maximum PC, Digital Trends, Android Authority, How-To Geek, Lifewire, and others.
-
ohim I`m confused, so... technically Samsung is "cheating" because they allow their CPU to run at full speed sometimes? ... that`s a "shocker". And i still don`t get it why would that CPU idle 3 cores when running a benchmark .. isn`t that application suppose to max out the hardware?Reply
-
Disingenuous Samsung is "cheating" because they purposefully created a bit a code that checks for the identity of the app that is running. That bit of code has a hard coded list of names that it checks to see if the app that is running is a popular benchmarking app. When it detects that it is, it behaves differently by going full throttle on its cpu core speeds to affect the outcome of the benchmark scores even before the benchmark is running.Reply
This is disingenuous since the performance seen on the benchmark is not truly what you can expect in real-world situations. Instead, the score has been overinflated thanks to that bit of code detecting the benchmarking app, thereby making people think that the device is more powerful than it truly is in real-world situations. -
qlum In short next time you benchmark phones make sure to modify the benchmark name first to get a fair score.Reply -
heero yuy so "benchmark mode" is where the processor runs at maximum and shows what the phone can do when on full beltReply
and that's cheating? I thought a benchmark is to see what the device can do when pushed -
laseru If you don't understand the basics better not write anything here. First it cheats because the phone in real life does not work like that. I hope you really understand that, it does not work like that in real life. If they do that at least they should say, but I am very curious why they don't do that with a game???? Because the battery which is very bad will be even worst maybe? Second the SoC on this phone the Snapdragon 800 has the same features as Turbo Boost, HyperThreading is not available here! Except that with Intel they don't need to keep all the cores online and at full speed when the pc is doing nothing just rendering the interface of a benchmark app. They should copy Intel and allow this boost of frequency on the fly but it seems for whatever reason if they do that the values are lower so that's the reason why they turn the CPU on and keep the cores running at 2,3Ghz doing nothing. Either way a cheap brand, cheap products(quality wise), cheap customization and add the regional lock to that and a high price. But there are people stupid enough to call this a premium device instead of a big scream device with average performance and a cartoonish OS. Don't get me wrong I don't hate Android as I own both IOS and Android but some things need t be said the right way.Reply -
Muhibah Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing offReply -
Muhibah Thats the keyword, all the time. Its ok to turboboost to any real application all the time as u benefit from it but it is cheating if u only turbo boost on benchmark for the sake of showing offReply -
house70 Idiotic move on Samsung's part. The phone is plenty fast without crap like this.Reply
Esp. since benchmarks now matter to only the few that still place any value into them (I am referring to phones benchmarks). For daily use they're totally irrelevant, people could give a rat's a$$ about bench results, and all is running the way it's supposed to run. Why risk getting caught like this, when you know the OS is open and can (and will) be examined by everybody with a dog in it? -
teh_chem Reply
I'm not defending what's been done, but that's not the case here. Turbo is generally available to any application, and is a low-level CPU load and TDP-dependent operating; not a software/code-dependent thing that only allows turbo under very specific conditions and applications. Manipulating CPU response in the way that Samsung is allegedly doing just makes benchmark comparisons fairly worthless--even more than they already were for extracting real-world usage experience.11639934 said:Thats the keyword, all the time. Its ok to turbobooat to any real application all the tine but it is cheating if u only turbo boost on benchmark for the sake of showing off