Android benchmark inflating seems to be common practice except for Apple, Motorola, Nvidia and Google.
Right after Ars Technica reported that Samsung was initiating a benchmark booster in the Galaxy Note 3 when specific applications and benchmarks are run, AnandTech reports that HTC, LG and Asus are also implementing a CPU boost on their smartphones and tablets. Even more, nearly all Android device makers have participated in benchmark inflation over the past several years.
AnandTech's report actually extends on the earlier report made back in July that Samsung was inflating benchmark performance on the Galaxy S4. Since then, the site has discovered that Samsung isn't the only one in the crowd, as stated above. In fact, the only companies not appearing to tamper with benchmarks is Apple, Motorola, Nvidia Shield, and Google with its Nexus devices.
"We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop," AnandTech reports. "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior."
"It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung," the site adds. "None of the Nexus do, which is understandable since the optimization isn’t a part of AOSP. This also helps explain why the Nexus 4 performed so slowly when we reviewed it – this mess was going on back then and Google didn’t partake."
Smartphones affected by benchmark tampering include the Asus Padfone Infinity, the HTC One, the HTC One Mini, the LG G2, the Samsung Galaxy S 4, the Samsung Galaxy Note 3, and the Samsung Galaxy Tab 3 10.1 and 2014 Edition tablets.
The report points out that the gains in performance granted by the artificial boost are not worth all the press these OEMs are receiving. The inflated scores provide up to 5 percent gains on the GPU front and up to 10 percent on the CPU front. Device makers thus should demand better performance and power efficiency from silicon vendors than play the benchmark wars with competing ODMs.
"Whether the OEMs choose to change or not however, we’ve seen how this story ends," the site states. "We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years."
What's funny is that Samsung is spearheading a consortium to provide more effective hardware and system-level performance assessment of mobile devices. Called MobileBench, this group includes not only Samsung, but Broadcom, Huawei, OPPO and Spreadtrum. The consortium is actively seeking new members now.
When launched, MobileBench will supposedly evaluate the performance of core mobile device elements such as application processors, popular memory solutions, embedded storage (eMMC) and graphics memory chips. MobileBench-UX will test user scenarios like switching among different apps, video shooting and viewing, and phone camera operations.