GFXBench 3.0: A Fresh Look At Mobile Benchmarking

A Much Needed Benchmark, Just In Time

We're dialed in with benchmarks on the PC. Our CPU suite is mix of single- and multi-threaded real-world apps, some of which exploit OpenCL acceleration to represent as wide of a range of workloads as possible. Our graphics testing involves the latest games, evaluated using uncompressed video captures of performance. The rigs we have set up are really quite cool. And the storage data we generate can get very low-level.

But the benchmarks we run on mobile devices are nowhere near as mature, and that includes GFXBench. As software developers push the boundaries of what they can do in these more closed-up operating environments, we're slowly seeing better tools to help us characterize the performance of smartphones and tablets across Android, iOS, Windows RT, and Windows 8.1. What we're finding along the way is that, as in the early days of PC benchmarking, a lot of companies are intentionally gaming tests for better results.

GFXBench 3.0 goes the furthest of any mobile-oriented metric we've used thus far to help isolate graphics performance, graphics quality, and battery life in graphics workloads. The ability to compare all three vectors in one story makes the decision to increase clock rates or alter output fidelity tougher for manufacturers to make. Of course, GFXBench 3.0 isn't foolproof. There's nothing stopping those same companies from pushing higher numbers in other titles and then playing more conservative in this test, where they know they'll be evaluated more broadly. Even still, we appreciate the robust comparisons we can make in GFXBench 3.0, along with the advanced API support and low-level components that help drill down into a GPU's strengths and weaknesses.

Take our journey with Samsung's Galaxy Note 10.1” 2014 Edition as an example. It wasn't until we ran the Render Quality tests that we were able to more thoroughly able to explain why the tablet might have taken a beating. Previously, our only conclusion would have been that ARM's Mali-T628MP6 just wasn't as fast. Now we can add that graphics horsepower is spent on a closer-to-reference 3D rendering, whereas competing products are taking quality shortcuts in the name of better frame rates.

Armed with this information, the concept of cheating gets a little more interesting. There really is a difference between legitimately increasing performance in a taxing workload at the expense of power (or maybe even understanding that a device is going to get hotter than it should), which some companies do, versus targeting benchmarks for big numbers, leaving real-world apps running slower to prolong battery life or prevent debilitating throttling. The practice of increasing voltages and frequencies upon app detection are already well-known, so it stands to reason that SoC vendors will try to make themselves look better in other ways. Messing with render quality is an approach we've seen from AMD and Nvidia on the desktop, and now we have a tool that can test for this on the mobile side, too.

Really, what GFXBench 3.0 lets us do is figure out where devices might be “robbing Peter to pay Paul”. You can turn the dial up and get bigger frame rates, but hurt battery life. GPUs can do the same by making the workload easier, hurting image quality. You can shoot for all-day longevity at the expense of speed. Or, you can output a great-looking image, burning cycles that could have gone to performance. We can now tell you how the balance is struck on a per-device basis.

And obviously, GFXBench 3.0 gives us an OpenGL ES 3.0-class workload to test modern SoCs with. The latest Adreno, Mali, PowerVR, Vivante, and Nvidia GPUs all support the API's enhanced features, paving the way for better-looking mobile games. But this is the first benchmark that lets us test performance under OpenGL ES 3.0. For that reason alone, it's an important component of our mobile benchmark suite.

Create a new thread in the US Reviews comments forum about this subject
This thread is closed for comments
21 comments
    Your comment
  • Cryio
    I was just about to write "why not WP", but then I remember WP games run on DirectX.
    0
  • panzerknacker
    Its cool u guys put so much effort into this but tbh most of the benchmark results seem to be completely random. Phones with faster SoC's performing slower and vice versa. I think there is no point at all benching a phone because 1. The benchmarking software is a POS and unreliable and 2. The phone OS's and apps are all complete POSs and act completely random in all kinda situations. I'd say just buy the phone with a fast SoC that looks the best to u and when it starts acting like a POS (which they all start doing in the end) buy a new one.
    -2
  • Marcus Wandle
    You show those dumb nay sayers, Apple.
    -1
  • umadbro
    What kind of bs is this? Force 720p on all devices and you'll see what happens to your precious 5s. Even my Zl murdered it.
    -6
  • andreluizbarbieri
    Why No mention about MX3 and Note beat iphone 5s?
    -2
  • jamsbong
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    -1
  • rolli59
    Well I have a smart phone but that is so I can receive business emails on the go, I have a tablet because it is great for watching movies on the go. Do I want to find out if there are any faster devices to do those things, not really while what I got is sufficient. I leave all the heavy tasks to the computers.
    -1
  • Durandul
    Quote:
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    If those are the only two benchmarks relevant to you, then I wonder why you are using a phone and not a 3DS or something. But seriously, most of the other devices have more than a million more pixels then the iPhone, so this benchmark is not so telling. It was mentioned before, but it would be nice to test at a given resolution, although as suppose applications don't give you an option on the phone.
    2
  • umadbro
    Quote:
    Quote:
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    If those are the only two benchmarks relevant to you, then I wonder why you are using a phone and not a 3DS or something. But seriously, most of the other devices have more than a million more pixels then the iPhone, so this benchmark is not so telling. It was mentioned before, but it would be nice to test at a given resolution, although as suppose applications don't give you an option on the phone.
    It does give the option to force some specific resolution. Don't know why this "review" didn't do it. That's what I've been trying to say from the start.
    2
  • umadbro
    Quote:
    Quote:
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    If those are the only two benchmarks relevant to you, then I wonder why you are using a phone and not a 3DS or something. But seriously, most of the other devices have more than a million more pixels then the iPhone, so this benchmark is not so telling. It was mentioned before, but it would be nice to test at a given resolution, although as suppose applications don't give you an option on the phone.
    It does give the option to force some specific resolution. Don't know why this "review" didn't do it. That's what I've been trying to say from the start.
    0
  • dragonsqrrl
    Anonymous said:
    Quote:
    Quote:
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    If those are the only two benchmarks relevant to you, then I wonder why you are using a phone and not a 3DS or something. But seriously, most of the other devices have more than a million more pixels then the iPhone, so this benchmark is not so telling. It was mentioned before, but it would be nice to test at a given resolution, although as suppose applications don't give you an option on the phone.
    It does give the option to force some specific resolution. Don't know why this "review" didn't do it. That's what I've been trying to say from the start.

    ... you guys realize that the off-screen tests render at 1080p, right? That's the whole point, to make direct performance comparisons regardless of a devices display resolution. It's also explained in the performance results.

    On a different note, I find it amazing how consistently and predictably the community on this site tries to discredit an objective review when the performance results favor an Apple device in any way. This isn't exactly breaking news for anyone who's familiar with SOC performance. Please try to set aside your childish biases and just accept the results for what they are. The A7 is a powerful SOC, get over it.
    4
  • h2323
    Looks like imagination techs powervr and the radeon..I mean adreno own.
    0
  • umadbro
    Quote:
    Anonymous said:
    Quote:
    Quote:
    The only relevant benchmarks are the first two because they are full-fletch 3D graphics, which is won by the most portable device; The iPhone. The rest of the benchies are just primitive 2D graphics which is irrelevant. Android devices won all those in flying colours.
    If those are the only two benchmarks relevant to you, then I wonder why you are using a phone and not a 3DS or something. But seriously, most of the other devices have more than a million more pixels then the iPhone, so this benchmark is not so telling. It was mentioned before, but it would be nice to test at a given resolution, although as suppose applications don't give you an option on the phone.
    It does give the option to force some specific resolution. Don't know why this "review" didn't do it. That's what I've been trying to say from the start.
    ... you guys realize that the off-screen tests render at 1080p, right? That's the whole point, to make direct performance comparisons regardless of a devices display resolution. It's also explained in the performance results.On a different note, I find it amazing how consistently and predictably the community on this site tries to discredit an objective review when the performance results favor an Apple device in any way. This isn't exactly breaking news for anyone who's familiar with SOC performance. Please try to set aside your childish biases and just accept the results for what they are. The A7 is a powerful SOC, get over it.
    I love people like you who claim to be some proffesinal "SoC experts" online while we the rest of us don't know nothing.A7 is a powerful SoC but the GPU is the same powervr as many other devices have. Apple tweaked the cpu cores only (at least going by the news).You talk about the off-screen tests which show exactly that the Android powered devices clearly pull back into the game with the A7, only one's the iPhone gets are the on screen tests which the iPhone runs at 720 and the androids run at 1080 - million(s) of more pixels to process then of course the load on the gpu is bigger therefor slower results. That's exactly why this review needs to force 720 on everything which you can easily do in the app itself on android.The two cores in an A7 are tweaked so much it keeps up with quad-core SD's etc, I got to give them that. But don't come rushing in telling that you know it all and everyone else are just dumb.
    -1
  • nebun
    apple has a good phone with the iPhone5s....not bad, considering how small it is....powerful indeed
    0
  • MANOFKRYPTONAK
    Why don't they do a chart that includes scores changed to the same resolution? I understand why they have these original charts that show performance on the devices screen, but why don't they have a chart with that shows real hardware performance?
    -1
  • lockhrt999
    Nexus 5 is slowest example of snapdragon 800. Why didn't you use Note 3?
    -1
  • daglesj
    ART or Dalvik on the KitKat Androids?
    0
  • daglesj
    Hmm well just did tests of the first two benches with my stock Nexus 4 running ART runtime and the scores were - 525 for Colossus1388 for TRex.Virtually the same as the Nexus 5 with a slower phone.Right okayyyy..
    0
  • daglesj
    The low levels were - ALU at 1800 / Alpha at 4534 / Driver at 406 / Fill at 2672.So looks like running ART can be a boost for older Android kit maybe.This was a straight install of the benchmark. No rebooting or shutting down of other apps and services.
    0
  • Ninjawithagun
    Fundamentally flawed benchmarking because the resolutions were not taken into consideration. Of course the Apple A7 processor is able to beat all the other processors because it has the least amount of graphics processing overhead. In comparison, the Nexus 7 @ 1920 x 1080 has to process 2.85 times (or 285%) more pixels per clock cycle versus the Apple A7 (1136 x 640). The benchmark scores would be nearly even (or even worse) if Apple A7 processor had to output a simliar resolution.
    -1