Skip to main content

How We Test Smartphones And Tablets

Benchmark Suite

The mixture of synthetic and real-world tests we run are meant to give a comprehensive overview of a device’s performance. Synthetic tests—which are generally composed of several small, specialized blocks of code for performing operations such as cryptography, file compression, matrix operations, and alpha blending—are good at isolating the performance of different parts of a component’s architecture, including integer and floating point math, specialized instructions, pixel shaders, and rasterization. With this information, we can make comparisons between individual hardware components like SoCs (CPU A is faster than CPU B) or NAND flash. Because of their highly focused nature, however, it can be difficult to relate these results to the overall user experience, limiting us to generic statements about a device being faster because it has a faster CPU in certain benchmarks. Furthermore, synthetic tests are generally designed to push hardware to its limits—useful for determining maximum performance and for spotting weak points in a design—but do not represent real-world workloads.

For this reason, we also try to include benchmarks that test macro-level activities you do every day such as web browsing, composing text, editing photos, or watching a video. While these benchmarks are a better indicator of overall user experience, they are much more difficult to develop, leaving testers few options for mobile platforms.

To truly understand the performance of a device, we need to test it at the component level and at the system level, we need to know its maximum performance and its performance in real-world scenarios, and we also need to spot deficiencies (thermal throttling) and anomalies (unsupported features). No single benchmark can do all of these things. There’s not even a single benchmark that can adequately test any one of these things (creating a good benchmark is extremely difficult and there are always compromises). This is why we run a whole suite of benchmarks, many of which have overlapping functionality.

By now it should be apparent that the benchmarks we use are not randomly selected. In addition to fulfilling the requirements above, our benchmark suite comes from experienced developers who are willing to openly discuss how their benchmarks work. We work closely with most of these developers so that we may gain a better understanding of the tests themselves and to provide them with feedback for improving their tests. The table below lists the benchmarks we currently use to test mobile devices.

Google Android

CategoryBenchmarkVersionDeveloper
CPU And System PerformanceAndEBench Pro 20152.1.2472EEMBC
Basemark OS II Full2.0Basemark Ltd
Geekbench 33.3.1Primate Labs
MobileXPRT 20131.0.92.1Principled Technologies
PCMark1.1Futuremark
TabletMark 20143.0.0.63BAPCo
Browsermark2.1Basemark Ltd
JSBench2013.1Purdue University
Google Octane2.0Google
Peacekeeper-Futuremark
GPU And Gaming Performance3DMark: Ice Storm Unlimited1.2Futuremark
Basemark X1.1Basemark Ltd
GFXBench 3 Corporate3.0.28Kishonti
GFXBench 3.1 Corporate3.1.0Kishonti
Basemark ES 3.11.0.2Basemark Ltd
Battery Life And Thermal ThrottlingBasemark OS II Full2.0Basemark Ltd
GFXBench 3 Corporate3.0.28Kishonti
PCMark1.1Futuremark
TabletMark 20143.0.0.63BAPCo

Apple iOS

CategoryBenchmarkVersionDeveloper
CPU And System PerformanceBasemark OS II Full2.0Basemark Ltd
Geekbench 33.3.4Primate Labs
TabletMark 20143.0.0.63BAPCo
Browsermark2.1Basemark Ltd
JSBench2013.1Purdue University
Google Octane2.0Google
Peacekeeper-Futuremark
GPU And Gaming Performance3DMark: Ice Storm Unlimited1.2Futuremark
Basemark X1.1Basemark Ltd
GFXBench 3 Corporate3.0.32Kishonti
GFXBench 3.1 Corporate3.1.0Kishonti
Basemark ES 3.11.0.2Basemark Ltd
Battery Life And Thermal ThrottlingBasemark OS II Full2.0Basemark Ltd
GFXBench 3 Corporate3.0.32Kishonti
TabletMark 20143.0.0.63BAPCo

Microsoft Windows Phone

CategoryBenchmarkVersionDeveloper
CPU And System PerformanceBasemark OS II Full2.0Basemark Ltd
Browsermark2.1Basemark Ltd
JSBench2013.1Purdue University
Google Octane2.0Google
Peacekeeper-Futuremark
GPU And Gaming PerformanceBasemark X1.1Basemark Ltd
GFXBench 3 DirectX3.0.4Kishonti
Battery Life And Thermal ThrottlingBasemark OS II Full2.0Basemark Ltd
GFXBench 3 DirectX3.0.4Kishonti