HTC, LG and Asus Also Caught Inflating Mobile Benchmarks

News
By Kevin Parrish
published

Android benchmark inflating seems to be common practice except for Apple, Motorola, Nvidia and Google.

Right after Ars Technica reported that Samsung was initiating a benchmark booster in the Galaxy Note 3 when specific applications and benchmarks are run, AnandTech reports that HTC, LG and Asus are also implementing a CPU boost on their smartphones and tablets. Even more, nearly all Android device makers have participated in benchmark inflation over the past several years.

AnandTech's report actually extends on the earlier report made back in July that Samsung was inflating benchmark performance on the Galaxy S4. Since then, the site has discovered that Samsung isn't the only one in the crowd, as stated above. In fact, the only companies not appearing to tamper with benchmarks is Apple, Motorola, Nvidia Shield, and Google with its Nexus devices.

"We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop," AnandTech reports. "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior."

"It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung," the site adds. "None of the Nexus do, which is understandable since the optimization isn’t a part of AOSP. This also helps explain why the Nexus 4 performed so slowly when we reviewed it – this mess was going on back then and Google didn’t partake."

Smartphones affected by benchmark tampering include the Asus Padfone Infinity, the HTC One, the HTC One Mini, the LG G2, the Samsung Galaxy S 4, the Samsung Galaxy Note 3, and the Samsung Galaxy Tab 3 10.1 and 2014 Edition tablets.

The report points out that the gains in performance granted by the artificial boost are not worth all the press these OEMs are receiving. The inflated scores provide up to 5 percent gains on the GPU front and up to 10 percent on the CPU front. Device makers thus should demand better performance and power efficiency from silicon vendors than play the benchmark wars with competing ODMs.

"Whether the OEMs choose to change or not however, we’ve seen how this story ends," the site states. "We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years."

What's funny is that Samsung is spearheading a consortium to provide more effective hardware and system-level performance assessment of mobile devices. Called MobileBench, this group includes not only Samsung, but Broadcom, Huawei, OPPO and Spreadtrum. The consortium is actively seeking new members now.

When launched, MobileBench will supposedly evaluate the performance of core mobile device elements such as application processors, popular memory solutions, embedded storage (eMMC) and graphics memory chips. MobileBench-UX will test user scenarios like switching among different apps, video shooting and viewing, and phone camera operations.

Kevin Parrish
15 Comments Comment from the forums
  • the1kingbob
    Might explain why the LG optimus G CPU benchmarks higher than the nexus 4 even though the hardware is exactly the same. (Given same version of android, I think 4.3 gave the N4 a boost)
    Reply
  • firefoxx04
    Who cares? Inflating or lying about benchmarks is one thing, the device actually pumping out those bench results is another. If the cpu runs at its max clock on all cores and scores higher than a similar device with the same CPU who cares?? That just means that the one device will run its self full throttle while the other might have a quad core but it never clocks up or enables all the cores and is therefor SLOWER.

    If my intel i7 benches higher than an identical i7 in a single threaded app because mine has turbo boost enabled makes me a cheater?
    Reply
  • jimmysmitty
    To me it's not an issue. Only because I already dismiss most of the benchmarks since they are synthetic and only tell you what the max it could do on that OS with that hardware at those specs.

    Its the same with desktop CPUs and GPUs. Synthetics are an example of a perfect world situation.

    To know if a device is right for you, the best way is to actually use the product and see.

    The S4 is a great phone. Considering that they all use similar hardware, then its all up to brand preference and a few other features like screen type and size etc.
    Reply
  • rwinches
    Where are all the indignant ratchetjaw commenters ready to crucify Samsung, now?
    Samsung may have discovered what others were doing and decided to work on a solution, MobileBench.
    Reply
  • BryanFRitt
    How about benchmarking the devices until the battery runs out, and report the results about battery life, etc...
    Reply
  • Vladislaus
    Why did they left Sony devices out of this test?
    Reply
  • Bolts Romano
    How many of you cares about the benchmark seriously. Not me , i think only few people. This is a mobile phone , not a workstation or game rigs Puhleease
    Reply
  • sykozis
    11647605 said:
    Who cares? Inflating or lying about benchmarks is one thing, the device actually pumping out those bench results is another. If the cpu runs at its max clock on all cores and scores higher than a similar device with the same CPU who cares?? That just means that the one device will run its self full throttle while the other might have a quad core but it never clocks up or enables all the cores and is therefor SLOWER.

    If my intel i7 benches higher than an identical i7 in a single threaded app because mine has turbo boost enabled makes me a cheater?

    You're completely missing the point here. These phones are running code that forces the CPU to run at full clocks on all cores, only when benchmarking apps are running. This isn't typical behavior for the processors and the phones do in fact perform very similarly under "normal" usage conditions. Normally, the phone will only clock up the cores necessary to complete the task it's performing and not necessarily run those cores at full clock speed. The manufacturers are actually tampering with how the processor is designed to function, to get higher benchmark scores.
    Reply
  • sykozis
    11647605 said:
    Who cares? Inflating or lying about benchmarks is one thing, the device actually pumping out those bench results is another. If the cpu runs at its max clock on all cores and scores higher than a similar device with the same CPU who cares?? That just means that the one device will run its self full throttle while the other might have a quad core but it never clocks up or enables all the cores and is therefor SLOWER.

    If my intel i7 benches higher than an identical i7 in a single threaded app because mine has turbo boost enabled makes me a cheater?

    You're completely missing the point here. These phones are running code that forces the CPU to run at full clocks on all cores, only when benchmarking apps are running. This isn't typical behavior for the processors and the phones do in fact perform very similarly under "normal" usage conditions. Normally, the phone will only clock up the cores necessary to complete the task it's performing and not necessarily run those cores at full clock speed. The manufacturers are actually tampering with how the processor is designed to function, to get higher benchmark scores.
    Reply
  • sykozis
    time to fix the posting issue....
    Reply