Benchmark Cheaters Galaxy S4, Note 3 Absolved in 4.4 Update

Last fall, several mobile phone manufacturers were caught cheating in Android benchmarks. HTC, LG, Asus, and Samsung were all found to be boosting clock rates when the devices ran specific benchmarks. In response, Futuremark delisted offending devices from its 3DMark performance rankings.

"That decision won us lots of praise from 3DMark users, even from those who owned one of the delisted models," said Jani Joki, director of engineering at Futuremark.

In light of the latest Android 4.4 KitKat update, Futuremark informed Tom's Hardware that the Samsung Galaxy Note III and Galaxy S4 now conform to its benchmark guidelines.

"We knew that delisting would send a message to manufacturers too. We hoped it would encourage them to be fair and honest with their customers," Joki added. "I am happy to say that our actions have paid off."

Samsung Galaxy S4

  • 3DMark scores are valid when using Android 4.2.2, 4.4.x or later*
  • Delisted when using Android 4.3 or 4.3.1

Samsung Galaxy Note III

  • 3DMark scores are valid when using Android 4.4.2 or later*
  • Delisted when using Android 4.3

As a result, Galaxy S4 and Galaxy Note III devices are now back on 3DMark's device channel.

We ran some tests of our own to see how Samsung Galaxy S4 performance is affected when going from Jelly Bean to KitKat.

In terms of raw score, we're pleased to see that there are some real performance gains in Android 4.4 KitKat. While 4.3 Jelly Bean still leads the pack in overall score, those numbers are tainted by an artificially-boosted clock frequency. In fact, the scores from 4.4 KitKat show an improvement in graphics performance over both 4.2.2 and the cheating 4.3.

Our testing led us to discover another interesting Android 4.4 KitKat-related performance quirk: the Samsung Galaxy S4 ROM throttles the SoC much more aggressively than it did in 4.2 and 4.3, making 4.4 a completely differently performing animal.

Follow Marcus Yam @MarcusYam. Follow us @tomshardware, on Facebook and on Google+.

Marcus Yam
Marcus Yam served as Tom's Hardware News Director during 2008-2014. He entered tech media in the late 90s and fondly remembers the days when an overclocked Celeron 300A and Voodoo2 SLI comprised a gaming rig with the ultimate street cred.
  • Spectre694
    Awesome glad to see at least Samsung is getting the message.
    Reply
  • Christopher Shaffer
    Why are you benchmarking a Google tablet and a no-brand phone against the Samsung Galaxy phones?Pretty irrelevant comparison.
    Reply
  • Marcus Yam
    Why are you benchmarking a Google tablet and a no-brand phone against the Samsung Galaxy phones?Pretty irrelevant comparison.
    I disagree. Completely relevant comparison as all listed devices run the Qualcomm Snapdragon 600.
    Reply
  • ferooxidan
    @Christopher: irrelevant? same spec, same cpu, same SoC? nice april fool's day joke
    Reply
  • Christopher Shaffer
    13010135 said:
    Why are you benchmarking a Google tablet and a no-brand phone against the Samsung Galaxy phones?Pretty irrelevant comparison.
    I disagree. Completely relevant comparison as all listed devices run the Qualcomm Snapdragon 600.

    With completely different screens and other hardware. The point is that the comparison should be between *phones* with similar hardware.

    Either way, at the end of the day it's a apples to oranges because the OS is the performance factor: raw Android on a Google tab vs. Galaxy-ware on Samsung, that's all I'm saying.

    There are other phones with a Snapdraon 600, too.
    Reply